$i7 dZddlZddlmZmZddlmZmZddlmZddl m Z ddl m Z dd l m Zdd l mZejd Zgd ZGd dZdedejjjdefdZdedefdZGddZdS)z)[Experimental] Text Only Local Tokenizer.N)AnyIterable)OptionalUnion)sentencepiece_model_pb2)_common)_local_tokenizer_loader) _transformers)typeszgoogle_genai.local_tokenizer)_parse_hex_byte_token_str_to_bytesLocalTokenizer_TextsAccumulatorceZdZdZddZdeefdZdeej ddfdZ dej ddfd Z d ej ddfd Z d ejdejfd ZdeejddfdZdeejddfdZdejddfdZdejdejfdZdejdejfdZdeeefdeeeffdZdedefdZdS)raAccumulates countable texts from `Content` and `Tool` objects. This class is responsible for traversing complex `Content` and `Tool` objects and extracting all the text content that should be included when calculating token counts. A key feature of this class is its ability to detect unsupported fields in `Content` objects. If a user provides a `Content` object with fields that this local tokenizer doesn't recognize (e.g., new fields added in a future API update), this class will log a warning. The detection mechanism for `Content` objects works by recursively building a "counted" version of the input object. This "counted" object only contains the data that was successfully processed and added to the text list for tokenization. After traversing the input, the original `Content` object is compared to the "counted" object. If they don't match, it signifies the presence of unsupported fields, and a warning is logged. returnNcg|_dSN_textsselfs f/var/www/html/jarvis.com/web/backend/venv/lib/python3.11/site-packages/google/genai/local_tokenizer.py__init__z_TextsAccumulator.__init__;s DKKKc|jSrrrs r get_textsz_TextsAccumulator.get_texts>s ;rcontentsc:|D]}||dSr) add_content)rrcontents r add_contentsz_TextsAccumulator.add_contentsAs2   w  rr!ctjg|j}|jr|jD]}|jJtj}|j|jtd|j |j|_|j &| |j |j |_ |j &| |j |j |_ |j +|j |_ |j|j |j||d|dkr#t"d|d|ddSdS)N)partsrolez6LocalTokenizers do not support non-text content types.T) exclude_nonezHContent contains unsupported types for token counting. Supported fields z. Got .)r Contentr%r$Part file_data inline_data ValueErrorvideo_metadata function_calladd_function_callfunction_responseadd_function_responsetextrappend model_dumploggerwarning)rr!counted_contentpart counted_parts rr z_TextsAccumulator.add_contentEsm"7<@@@O}3-33$$000z|| > %)9)EF      *(,(;, %   )  !3 4 4 4'+'9, $  ! -  $ $T%; < < <+/+A, ( 9 "i,  +  TY ' ' '$$\2222t,,0J0J1K11 nn 7$ 7 7,3 7 7 7rr.c|jr|j|jtj|j}|jr#||j}||_dSdS)zProcesses a function call and adds relevant text to the accumulator. Args: function_call: The function call to process. )nameN)r;rr3r FunctionCallargs_dict_traverse)rr.counted_function_call counted_argss rr/z#_TextsAccumulator.add_function_calldsy - k+,,,!.M4FGGG0(();<)rr0counted_function_responsecounted_responses rr1z'_TextsAccumulator.add_function_responses!& 6 8 8> k*/000'8'=$!<,,->-GHH+;(((<#>  D k-2333*>*C"'' k-9:::  *#.&C??+?+JKK0B"-$?)=)FGG.>"+ ''rschemactj}|jr |j|_|jr |j|_|j |j|_|jr+|j|j|j|_|jr+|j|j|j|_|j r+|j |j |j |_ |j r+|j |j |j |_ |j r |j |_ |j r!||j }||_ |jr\i}|j D]9\}}|j|||}|||<:||_|jr!||j}||_|S)zProcesses a schema and adds relevant text to the accumulator. Args: schema: The schema to process. Returns: The new schema object with only countable fields. )r Schematypetitledefaultformatrr3rWenumextendrequiredproperty_orderingitemsrY propertiesexample _any_traverse) rr[counted_schemacounted_schema_itemsdkeyvalue counted_valuecounted_schema_examples rrYz_TextsAccumulator.add_schemas\^^N {("Kn |*#\n ~!%~n }, k'''$mn 6 k+,,,#)#5n  {( k%%%"Kn 0 k))) &n B)/)An& |2!__V\::1n $ a)//11*#u 3.. #"#n ~6#11&.AA5n rrlci}|jt||D]\}}||||<|S)zProcesses a dict and adds relevant text to the accumulator. Args: d: The dict to process. Returns: The new dict object with only countable fields. )rrclistkeysrfri)rrl counted_dictrmvals rr>z _TextsAccumulator._dict_traversesiLKtAFFHH~~&&&GGII22S,,S11l3 rrnct|trj||St|tr|St|t rfd|DS|S)zProcesses a value and adds relevant text to the accumulator. Args: value: The value to process. Returns: The new value with only countable fields. c:g|]}|S)ri).0itemrs r z3_TextsAccumulator._any_traverse..s' 9 9 94d  && 9 9 9r) isinstancestrrr3dictr>rr)rrns` rriz_TextsAccumulator._any_traverses% k l E4    ' '' E4  9 9 9 95 9 9 99 lr)rN)__name__ __module__ __qualname____doc__rrr}rr r(r"r r<r/rDrIrLrQrOr1rVrEr]rYr~rr>rirxrrrr's &    # 8EM#: t    4> 0U-? 0D 0 0 0 0 5: %*    Xej1d4 ()? @4 4444 <$5 <  < < < <("'";(  (((((+u|+ ++++Z d38n c3h    rrtokenr^rc|tjjjjkr$t |ddS|dddS)Nrbig)length byteorderu▁ zutf-8) r ModelProto SentencePieceTypeBYTEr to_bytesreplaceencode)rr^s rrrs_ $ / = B GGG 5 ! ! * *!u * E EE == $ $ + +G 4 44rclt|dkrtd||dr|dstd| t |ddd}n #t$rtd |wxYw|d krtd ||S) zParses a hex byte string of the form '<0xXX>' and returns the integer value. Raises ValueError if the input is malformed or the byte value is invalid. zInvalid byte length: z<0x>zInvalid byte format: zInvalid hex value: zByte value out of range: )lenr, startswithendswithint)rrus rr r s  ZZ1__ 4U44 5 55   % 6s(;(;6 4U44 5 554 eAaCj"  CC 444 2522 3 334 CZZ 888 9 99 *s #A<<BceZdZdZdefdZejddddee j e j fde e j d e jfd Zejddee j e j fd e jfd ZdS) ra[Experimental] Text Only Local Tokenizer. This class provides a local tokenizer for text only token counting. LIMITATIONS: - Only supports text based tokenization and no multimodal tokenization. - Forward compatibility depends on the open-source tokenizer models for future Gemini versions. - For token counting of tools and response schemas, the `LocalTokenizer` only supports `types.Tool` and `types.Schema` objects. Python functions or Pydantic models cannot be passed directly. model_namectj||_tj|j|_tj|j|_dSr)loaderget_tokenizer_name_tokenizer_nameload_model_proto _model_protoget_sentencepiece _tokenizer)rrs rrzLocalTokenizer.__init__$sD!4Z@@D/0DEED.t/CDDDOOOrzThe SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.N)configrrrctj|}t}tj|pi}|||jr||j|j r+|j j r| |j j |j r-|tj|j g|j t|}tjt%d|DS)aCounts the number of tokens in a given text. Args: contents: The contents to tokenize. config: The configuration for counting tokens. Returns: A `CountTokensResult` containing the total number of tokens. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.count_tokens("What is your name?") print(result) # total_tokens=5 c34K|]}t|VdSr)r)rytokenss r z.LocalTokenizer.count_tokens..Rs(??V??????r) total_tokens)t t_contentsrr CountTokensConfigmodel_validater"rJrLgeneration_configresponse_schemarYsystem_instructionrrrrrCountTokensResultsum)rrrprocessed_contentstext_accumulator tokens_lists r count_tokenszLocalTokenizer.count_tokens)s1:h//(**  $ 3 3FLb A AF!!"4555 |/  ... LF$<$LL!!&":"JKKK  O##AL&2K1L$M$MNNN/((.>.H.H.J.J)K)KLLK  "??;?????   rc&tj|}t}|D]}||j|}g}|D]-}|jr$|jD]}||j .g}t||D]O\} } |tj d| j Dfd| j D| Ptj|S)a,Computes the tokens ids and string pieces in the input. Args: contents: The contents to tokenize. Returns: A `ComputeTokensResult` containing the token information. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.compute_tokens("What is your name?") print(result) # tokens_info=[TokensInfo(token_ids=[279, 329, 1313, 2508, 13], tokens=[b' What', b' is', b' your', b' name', b'?'], role='user')] cg|] }|j Srx)id)rypieces rr{z1LocalTokenizer.compute_tokens..sCCCeCCCrcng|]1}t|jjj|jj2Srx)rrrpiecesrr^)ryrrs rr{z1LocalTokenizer.compute_tokens..sL&k4#4#;EH#E#Jr) token_idsrr%) tokens_info)rrrr rEncodeAsImmutableProtorr$r3r%zipr TokensInforComputeTokensResult) rrrrr! tokens_protosroles_ token_infos tokens_protor%s ` rcompute_tokenszLocalTokenizer.compute_tokensUse4h//(**%,,""7++++O::""$$M E%%% % % %A ,,w| $ $ $ $K!-77   d  CC|/BCCC ,2         $ = = ==r)rrrrr}rr experimental_warningrr ContentListUnionContentListUnionDictrCountTokensConfigOrDictrrrrrxrrrrs  EEEEE  7B9= &&&e,e.HHI&u45 &   &&& &P 7B2>e,e.HHI2>  2>2>2> 2>2>2>rr)rloggingtypingrrrr sentencepiecerr r rr rr getLoggerr5__all__rr}rrrbytesrrr rrxrrrs 0/ """"""""111111//////   9 : :   MMMMMMMM`5 5-8FK5 5555 3 3    ,u>u>u>u>u>u>u>u>u>u>r