| Server IP : 209.209.40.120 / Your IP : 216.73.217.112 Web Server : Microsoft-IIS/10.0 System : Windows NT NEWWWW 10.0 build 17763 (Windows Server 2019) i586 User : NEWWWW$ ( 0) PHP Version : 8.3.30 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : OFF | Perl : OFF | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : C:/Python312/Lib/site-packages/pandas/tests/io/parser/__pycache__/ |
Upload File : |
�
0 �g� � �N � d Z ddlmZ ddlmZ ddlmZ ddlZddl Z ddl
Z ddl
m Z ddl
mZ ddlmZ e j$ j' d� Ze j$ j* e j$ j, gZe j$ j1 d � d
� � Zd � Zed � � Zy)
zd
Tests multithreading behaviour for reading and
parsing files for each parser defined in parsers.py
� )� ExitStack)�BytesIO)�
ThreadPoolN)� DataFrame)�Version�
pyarrow_xfailz0ignore:Passing a BlockManager:DeprecationWarningc �j �
� | }|j dk( ret j d� }t |j � t d� k r/|j t j j d�� � d�
d}�
fd�t |� D � }t � 5 }|D �cg c] }|j t |� � �� }}|j t d� � } | j |j |� }
|
d } |
D ] } t j | | � � d d d � y c c}w # 1 sw Y y xY w)
N�pyarrowz16.0z+# ValueError: Found non-unique column index)�reason�d �
c 3 � �K � | ]E }d j t �� D �cg c] }|d�d|d�d|d���� c}� j � �� �G yc c}w �w)�
�d�,N)�join�range�encode)�.0�_�i�
max_row_ranges ��JC:\Python312\Lib\site-packages\pandas/tests/io/parser/test_multi_thread.py� <genexpr>z7test_multi_thread_string_io_read_csv.<locals>.<genexpr>( sU �� �� � �
�
� � ��}�1E�F�A�a��U�!�A�a�5��!�A��'�F�G�N�N�P���F�s � A�A
� A� r )�engine�pytest� importorskipr � __version__� applymarker�mark�xfailr r �
enter_contextr r �map�read_csv�tm�assert_frame_equal)� all_parsers�request�parser�pa� num_files� bytes_to_df�stack�b�files�pool�results� first_result�resultr s @r �$test_multi_thread_string_io_read_csvr5 s �� � �F�
�}�}� �!�
�
�
��
+�� �2�>�>� "�W�V�_� 4� � � �� � �!�!�)V�!�W�
� �M��I���y�!��K�
�� 8��:E�F�Q��$�$�W�Q�Z�0�F��F��"�"�:�a�=�1���(�(�6�?�?�E�2�� �q�z� � � 8�F� � !� !�,�� 7� 8� 8� 8��F� 8� 8�s �D)� !D$� AD)�$D)�)D2c �6 � �� � �fd�}t |� D �cg c] }||z |z ||z f�� }}t |�� 5 }|j ||� }ddd� d j } |dd D ] }
| |
_ � t j
|� } | S c c}w # 1 sw Y �ExY w)a�
Generate a DataFrame via multi-thread.
Parameters
----------
parser : BaseParser
The parser object to use for reading the data.
path : str
The location of the CSV file to read.
num_rows : int
The number of rows to read per task.
num_tasks : int
The number of tasks to use for reading this DataFrame.
Returns
-------
df : DataFrame
c � �� | \ }}|s�j �dd|dg�� S �j �ddt |� dz |dg�� S )aj
Create a reader for part of the CSV.
Parameters
----------
arg : tuple
A tuple of the following:
* start : int
The starting row to start for parsing CSV
* nrows : int
The number of rows to read.
Returns
-------
df : DataFrame
r �date)� index_col�header�nrows� parse_datesN� � )r9 r: �skiprowsr; r< )r% �int)�arg�startr; r* �paths ��r �readerz0_generate_multi_thread_dataframe.<locals>.readerN sh �� �$ � ��u���?�?���!�5�v�h� #� �
� ��� �����Z�!�^����
�
�
� )� processesNr r= )r r r$ �columns�pd�concat) r* rC �num_rows� num_tasksrD r �tasksr1 r2 r: �r�final_dataframes `` r � _generate_multi_thread_dataframerO : s� �� �(
�F EJ�)�DT�
�?@��A� �� "�H� �$9�:�
�E�
�
�i� (� *�D��(�(�6�5�)��*� �Q�Z�
�
�F�
�Q�R�[� ���� �� �i�i��(�O� ���
�*� *�s �B
�B�Bc �$ � d}d}| }d}t t j j d� j |� t j j d� j |� t j j d� j |� t j j d� j |� t j j d� j |� dg|z dg|z dg|z t j
d|d �
� t j |d � � d
�
� }t j |� 5 }|j |� t ||||� }t j ||� d d d � y # 1 sw Y y xY w)N� �0 z__thread_pool_reader__.csv� �foo�bar�bazz20000101 09:00:00�s)�periods�freq�int64)�dtype)
�ar/ �cr �erT rU rV r8 r@ ) r �np�random� default_rngrH �
date_range�aranger& � ensure_clean�to_csvrO r' )r( rK rJ r* � file_name�dfrC rN s r �)test_multi_thread_path_multipart_read_csvrh � sJ � � �I��H�
�F�,�I� ����&�&�q�)�0�0��:����&�&�q�)�0�0��:����&�&�q�)�0�0��:����&�&�q�)�0�0��:����&�&�q�)�0�0��:��7�X�%��7�X�%��7�X�%��M�M�"5�x�c�R��9�9�X�W�5�
�
�B�
� � �� #� 3�t�
� � �$��:� �D�(�I�
�� � � �b�/�2�
3� 3� 3�s �6F�F) �__doc__�
contextlibr �ior �multiprocessing.poolr �numpyr_ r �pandasrH r �pandas._testing�_testingr& �pandas.util.versionr r! � usefixtures�
xfail_pyarrow�
single_cpu�slow�
pytestmark�filterwarningsr5 rO rh � rE r �<module>ry s� ��� !� � +� �
� � � � '�� � �'�'��8�
�
�K�K���
�K�K����
� ����� N�O�8� P�8�<C�L � 3� � 3rE