Skip to content

πŸ•·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

3.9 KB
Raw Blame

title: "μ μ‘ν˜• νŒŒμ‹±μ„ μœ„ν•œ μ €μž₯μ†Œ λ§Œλ“€κΈ°" description: "Scrapling의 μ μ‘ν˜• νŒŒμ‹±μ„ μœ„ν•œ save와 retrieveλ₯Ό κ΅¬ν˜„ν•˜μ„Έμš”. Redis μ €μž₯μ†Œλ₯Ό λ§Œλ“€μ–΄ μš”μ†Œ 속성을 μ €μž₯ν•˜κ³  κ°€μ Έμ˜΅λ‹ˆλ‹€."

μ €μž₯κ³Ό 검색 μ‹œμŠ€ν…œ λ§Œλ“€κΈ° {#writing-your-retrieval-system}

Scrapling은 기본적으둜 SQLiteλ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. 이 μ•ˆλ‚΄μ—μ„œλŠ” adaptive κΈ°λŠ₯에 ν•„μš”ν•œ μš”μ†Œ 속성을 직접 λ§Œλ“  μ €μž₯μ†Œμ— μ €μž₯ν•˜λŠ” 방법을 μ„€λͺ…ν•©λ‹ˆλ‹€.

예λ₯Ό λ“€μ–΄ Firebaseλ₯Ό μ‚¬μš©ν•΄ μ—¬λŸ¬ μ»΄ν“¨ν„°μ—μ„œ μ‹€ν–‰ν•˜λŠ” μŠ€νŒŒμ΄λ”κ°€ λ°μ΄ν„°λ² μ΄μŠ€λ₯Ό κ³΅μœ ν•˜λ„λ‘ λ§Œλ“€ 수 μžˆμŠ΅λ‹ˆλ‹€. 이런 온라인 λ°μ΄ν„°λ² μ΄μŠ€λ₯Ό μ“°λ©΄ μŠ€νŒŒμ΄λ” μ‚¬μ΄μ—μ„œ μ μ‘ν˜• 데이터λ₯Ό κ³΅μœ ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

μ €μž₯μ†Œ ν΄λž˜μŠ€λŠ” λ‹€μŒ μ„Έ κ°€μ§€ 쑰건을 λ§Œμ‘±ν•΄μ•Ό ν•©λ‹ˆλ‹€.

  1. 좔상 클래슀 scrapling.core.storage.StorageSystemMixin을 μƒμ†ν•˜κ³  λ¬Έμžμ—΄ 인수λ₯Ό λ°›μŠ΅λ‹ˆλ‹€. 라이브러리의 λ™μž‘μ„ μœ μ§€ν•˜κΈ° μœ„ν•œ url μΈμˆ˜μž…λ‹ˆλ‹€.
  2. 클래슀 μœ„μ— functools.lru_cache λ°μ½”λ ˆμ΄ν„°λ₯Ό μ‚¬μš©ν•΄ λ‹€λ₯Έ 클래슀처럼 Singleton νŒ¨ν„΄μ„ λ”°λ¦…λ‹ˆλ‹€.
  3. νƒ€μž… νžŒνŠΈμ— 맞게 save와 retrieveλ₯Ό κ΅¬ν˜„ν•©λ‹ˆλ‹€.
    • saveλŠ” 아무것도 λ°˜ν™˜ν•˜μ§€ μ•ŠμœΌλ©° λΌμ΄λΈŒλŸ¬λ¦¬μ—μ„œ 인수 두 개λ₯Ό λ°›μŠ΅λ‹ˆλ‹€.
      • 첫 λ²ˆμ§ΈλŠ” μš”μ†Œ 자체인 lxml.html.HtmlElementμž…λ‹ˆλ‹€. 같은 ν˜•μ‹μ„ μœ μ§€ν•˜λ €λ©΄ scrapling.core.utils._StorageTools ν•˜μœ„ λͺ¨λ“ˆμ˜ element_to_dict둜 λ”•μ…”λ„ˆλ¦¬λ‘œ λ°”κΎΌ λ’€ μ›ν•˜λŠ” λ°©μ‹μœΌλ‘œ λ°μ΄ν„°λ² μ΄μŠ€μ— μ €μž₯ν•˜μ„Έμš”.
      • 두 λ²ˆμ§ΈλŠ” 검색에 μ‚¬μš©ν•  λ¬Έμžμ—΄ μ‹λ³„μžμž…λ‹ˆλ‹€. μ΄ˆκΈ°ν™”ν•  λ•Œ 받은 urlκ³Ό 이 μ‹λ³„μžλ₯Ό μ‘°ν•©ν•œ 값은 각 ν–‰λ§ˆλ‹€ κ³ μœ ν•΄μ•Ό ν•©λ‹ˆλ‹€. κ·Έλ ‡μ§€ μ•ŠμœΌλ©΄ adaptive 데이터가 μ„žμž…λ‹ˆλ‹€.
    • retrieveλŠ” λ¬Έμžμ—΄ μ‹λ³„μžλ₯Ό λ°›μŠ΅λ‹ˆλ‹€. μ΄ˆκΈ°ν™”ν•  λ•Œ 받은 urlκ³Ό ν•¨κ»˜ μ‚¬μš©ν•΄ λ°μ΄ν„°λ² μ΄μŠ€μ—μ„œ μš”μ†Œμ˜ λ”•μ…”λ„ˆλ¦¬λ₯Ό μ°Ύκ³  λ°˜ν™˜ν•©λ‹ˆλ‹€. μ—†μœΌλ©΄ None을 λ°˜ν™˜ν•©λ‹ˆλ‹€.

더 μžμ„Έν•œ μ˜ˆκ°€ ν•„μš”ν•˜λ©΄ storage_adaptors 파일의 SQLite3 κ΅¬ν˜„μ„ μ‚΄νŽ΄λ³΄μ„Έμš”.

이 쑰건을 λ§Œμ‘±ν•˜λ©΄ λ‚˜λ¨Έμ§€λŠ” κ°„λ‹¨ν•©λ‹ˆλ‹€. μ—¬λŸ¬ μŠ€λ ˆλ“œλ₯Ό μ‚¬μš©ν•˜λŠ” 앱이라면 ν΄λž˜μŠ€κ°€ μ•ˆμ „ν•˜κ²Œ λ™μž‘ν•˜λŠ”μ§€ ν™•μΈν•˜μ„Έμš”. κΈ°λ³Έ μ €μž₯μ†Œ ν΄λž˜μŠ€λŠ” μŠ€λ ˆλ“œ μ•ˆμ „μ„±μ„ μ œκ³΅ν•©λ‹ˆλ‹€.

좔상 ν΄λž˜μŠ€μ—λŠ” μ‚¬μš©ν•  수 μžˆλŠ” 보쑰 ν•¨μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€. μ„€λͺ…이 μžμ„Ένžˆ μžˆλŠ” μ½”λ“œλ₯Ό μ‚΄νŽ΄λ³΄μ„Έμš” :)

μ‹€μ œ 예: Redis μ €μž₯μ†Œ {#real-world-example-redis-storage}

λ‹€μŒμ€ AI둜 λ§Œλ“  Redis μ €μž₯μ†Œ μ˜ˆμž…λ‹ˆλ‹€.

import redis
import orjson
from functools import lru_cache
from scrapling.core.storage import StorageSystemMixin
from scrapling.core.utils import _StorageTools

@lru_cache(None)
class RedisStorage(StorageSystemMixin):
    def __init__(self, host='localhost', port=6379, db=0, url=None):
        super().__init__(url)
        self.redis = redis.Redis(
            host=host,
            port=port,
            db=db,
            decode_responses=False
        )
        
    def save(self, element, identifier: str) -> None:
        # Convert element to dictionary
        element_dict = _StorageTools.element_to_dict(element)
        
        # Create key
        key = f"scrapling:{self._get_base_url()}:{identifier}"
        
        # Store as JSON
        self.redis.set(
            key,
            orjson.dumps(element_dict)
        )
        
    def retrieve(self, identifier: str) -> dict | None:
        # Get data
        key = f"scrapling:{self._get_base_url()}:{identifier}"
        data = self.redis.get(key)
        
        # Parse JSON if exists
        if data:
            return orjson.loads(data)
        return None