o 5c@sdZddlmZddlmZddlmZddlZddl Z ddl Z ddl m Z ddl mZe jdZdd Ze jjd d Zd d Ze jjddZdS)zd Tests multithreading behaviour for reading and parsing files for each parser defined in parsers.py ) ExitStack)BytesIO) ThreadPoolN) DataFrame pyarrow_skipcCs\ttj|dtdd}d|d<d|d<d|d<tjd|dd |d <tj|d d |d <|S)z Construct a DataFrame for testing. Parameters ---------- num_rows : int The number of rows for our DataFrame. Returns ------- df : DataFrame abcde)columnsfoobarbazz20000101 09:00:00s)periodsfreqdateint64)dtypeint)rnprandomrandlistpd date_rangearange)num_rowsdfr^/var/www/html/gps/gps/lib/python3.10/site-packages/pandas/tests/io/parser/test_multi_thread.py_construct_dataframes rc s|}dd}fddt|D}t/fdd|D}td}||j|}|d}|D]}t||q2WddS1sFwYdS)Ni'dcs(g|]}dddtDqS) cSs&g|]}|dd|dd|dqS)d,r.0irrr 3s&zCtest_multi_thread_string_io_read_csv...)joinrangeencode)r%_) max_row_rangerrr'2sz8test_multi_thread_string_io_read_csv..csg|] }t|qSr) enter_contextr)r%b)stackrrr'9sr)r)rr-rmapread_csvtmassert_frame_equal) all_parsersparser num_files bytes_to_dffilespoolresults first_resultresultr)r,r/r$test_multi_thread_string_io_read_csv+s "r>c sfdd}fddtD}td}|||}Wdn1s)wY|dj}|ddD]} || _q9t|} | S) a Generate a DataFrame via multi-thread. Parameters ---------- parser : BaseParser The parser object to use for reading the data. path : str The location of the CSV file to read. num_rows : int The number of rows to read per task. num_tasks : int The number of tasks to use for reading this DataFrame. Returns ------- df : DataFrame csB|\}}|sjdd|dgdSjddt|d|dgdS)aj Create a reader for part of the CSV. Parameters ---------- arg : tuple A tuple of the following: * start : int The starting row to start for parsing CSV * nrows : int The number of rows to read. Returns ------- df : DataFrame rr) index_colheadernrows parse_datesN )r?r@skiprowsrArB)r2r)argstartrA)r6pathrrreaderXs  z0_generate_multi_thread_dataframe..readercs g|] }|fqSrrr$)r num_tasksrrr'zsz4_generate_multi_thread_dataframe..) processesNrrC)r)rr1r rconcat) r6rHrrJrItasksr:r;r@rfinal_dataframer)rrJr6rHr _generate_multi_thread_dataframeDs "   rPcCsld}d}|}d}t|}t|}||t||||}t||WddS1s/wYdS)Niz__thread_pool_reader__.csv)rr3 ensure_cleanto_csvrPr4)r5rJrr6 file_namerrHrOrrr)test_multi_thread_path_multipart_read_csvs  "rU)__doc__ contextlibriormultiprocessing.poolrnumpyrpytestpandasrrpandas._testing_testingr3mark usefixtures pytestmarkrslowr>rPrUrrrrs        F