bes  Updated for version 3.20.10
dmrpp::Chunk Class Reference

#include <Chunk.h>

Collaboration diagram for dmrpp::Chunk:
Collaboration graph

Public Member Functions

void add_tracking_query_param ()
 Modify this chunk's data URL so that it includes tracking info. More...
 
 Chunk ()
 Get an empty chunk. More...
 
 Chunk (const Chunk &h4bs)
 
 Chunk (std::shared_ptr< http::url > data_url, std::string order, unsigned long long size, unsigned long long offset, const std::string &pia_str="")
 Get a chunk initialized with values. More...
 
 Chunk (std::shared_ptr< http::url > data_url, std::string order, unsigned long long size, unsigned long long offset, const std::vector< unsigned long long > &pia_vec)
 Get a chunk initialized with values. More...
 
 Chunk (std::string order, unsigned long long size, unsigned long long offset, const std::string &pia_str="")
 Get a chunk initialized with values, the data URL will not be set. More...
 
 Chunk (std::string order, unsigned long long size, unsigned long long offset, const std::vector< unsigned long long > &pia_vec)
 Get a chunk initialized with values, the data URl will not be set. More...
 
virtual void dump (std::ostream &strm) const
 
virtual void filter_chunk (const std::string &filters, unsigned long long chunk_size, unsigned long long elem_width)
 filter data in the chunk More...
 
virtual std::string get_byte_order ()
 Get the chunk byte order. More...
 
virtual unsigned long long get_bytes_read () const
 Get the number of bytes read so far for this Chunk. More...
 
virtual std::string get_curl_range_arg_string ()
 Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream. More...
 
virtual std::shared_ptr< http::urlget_data_url () const
 Get the data url for this Chunk's data block. More...
 
virtual bool get_is_read ()
 
virtual unsigned long long get_offset () const
 Get the offset to this Chunk's data block. More...
 
virtual const std::vector< unsigned long long > & get_position_in_array () const
 
virtual char * get_rbuf ()
 
virtual unsigned long long get_rbuf_size () const
 
virtual std::string get_response_content_type ()
 Get the response type of the last response. More...
 
virtual unsigned long long get_size () const
 Get the size of this Chunk's data block on disk. More...
 
Chunkoperator= (const Chunk &rhs)
 
virtual void read_chunk ()
 
virtual void set_bytes_read (unsigned long long bytes_read)
 Set the size of this Chunk's data block. More...
 
virtual void set_data_url (std::shared_ptr< http::url > data_url)
 Set the data url for this Chunk's data block. More...
 
virtual void set_is_read (bool state)
 
void set_position_in_array (const std::string &pia)
 parse the chunk position string More...
 
void set_position_in_array (const std::vector< unsigned long long > &pia)
 Set the chunk's position in the Array. More...
 
virtual void set_rbuf_to_size ()
 Allocates the internal read buffer to be d_size bytes. More...
 
void set_read_buffer (char *buf, unsigned long long buf_size, unsigned long long bytes_read=0, bool assume_ownership=true)
 Set the target read buffer for this chunk. More...
 
void set_response_content_type (const std::string &ct)
 Set the response type of the last response. More...
 
virtual std::string to_string () const
 

Static Public Member Functions

static void parse_chunk_position_in_array_string (const std::string &pia, std::vector< unsigned long long > &pia_vect)
 

Protected Member Functions

void _duplicate (const Chunk &bs)
 

Friends

class ChunkTest
 
class DmrppCommonTest
 
class MockChunk
 

Detailed Description

This class is used to encapsulate the state and behavior needed for reading chunked data associated with a DAP variable. In particular it is based on the semantics of an hdf4:Chunk object, which is used to represent a chunk of data in a (potentially complex) HDF4/HDF5 file.

Definition at line 59 of file Chunk.h.

Constructor & Destructor Documentation

◆ Chunk() [1/5]

dmrpp::Chunk::Chunk ( )
inline

Get an empty chunk.

Note
This constructor does not read the Query String marker from the BES context system. You must call Chunk::add_tracking_query_param() if you want that information added with Chunks created using this constructor.
See also
Chunk::add_tracking_query_param()

Definition at line 135 of file Chunk.h.

◆ Chunk() [2/5]

dmrpp::Chunk::Chunk ( std::shared_ptr< http::url data_url,
std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::string &  pia_str = "" 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte_order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_strA string that provides the logical position of this chunk in an Array. Has the syntax '[1,2,3,4]'.

Definition at line 152 of file Chunk.h.

◆ Chunk() [3/5]

dmrpp::Chunk::Chunk ( std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::string &  pia_str = "" 
)
inline

Get a chunk initialized with values, the data URL will not be set.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte_order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_strA string that provides the logical position of this chunk in an Array. Has the syntax '[1,2,3,4]'.

Definition at line 185 of file Chunk.h.

◆ Chunk() [4/5]

dmrpp::Chunk::Chunk ( std::shared_ptr< http::url data_url,
std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::vector< unsigned long long > &  pia_vec 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_vecThe logical position of this chunk in an Array; a std::vector of unsigned ints.

Definition at line 216 of file Chunk.h.

◆ Chunk() [5/5]

dmrpp::Chunk::Chunk ( std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::vector< unsigned long long > &  pia_vec 
)
inline

Get a chunk initialized with values, the data URl will not be set.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_vecThe logical position of this chunk in an Array; a std::vector of unsigned ints.

Definition at line 250 of file Chunk.h.

Member Function Documentation

◆ add_tracking_query_param()

void dmrpp::Chunk::add_tracking_query_param ( )

Modify this chunk's data URL so that it includes tracking info.

Add information to the Query string of a URL, intended primarily to aid in tracking the origin of requests when reading data from S3. The information added to the query string comes from a BES Context command sent to the BES by a client (e.g., the OLFS). The addition takes the form "tracking_context=<context value>". The method checks to see if the URL already has a query string, if not it adds one: "?tracking_context=<context value>" And if so it appends an additional parameter: "&tracking_context=<context value>"

Note
This is only added to data URLs that reference an S3 bucket.

Cloudydap test hack where we tag the S3 URLs with a query string for the S3 log in order to track S3 requests. The tag is submitted as a BESContext with the request. Here we check to see if the request is for an AWS S3 object, if it is AND we have the magic BESContext "cloudydap" then we add a query parameter to the S3 URL for tracking purposes.

Should this be a function? FFS why? This is the ONLY place where this needs happen, as close to the curl call as possible and we can just turn it off down the road. - ndp 1/20/17 (EOD)

Well, it's a function now... ;-) jhrg 8/6/18

Definition at line 526 of file Chunk.cc.

◆ dump()

void dmrpp::Chunk::dump ( std::ostream &  strm) const
virtual

unsigned long long d_size; unsigned long long d_offset; std::string d_md5; std::string d_uuid; std::string d_data_url; std::vector<unsigned int> d_chunk_position_in_array;

Definition at line 880 of file Chunk.cc.

◆ filter_chunk()

void dmrpp::Chunk::filter_chunk ( const std::string &  filters,
unsigned long long  chunk_size,
unsigned long long  elem_width 
)
virtual

filter data in the chunk

This method tracks if a chunk has already been decompressed, so, like read_chunk() it can be called for a chunk that has already been decompressed without error.

Parameters
filtersSpace separated list of filters
chunk_sizeThe expected chunk size, in elements; used to allocate storage
elem_widthThe number of bytes per element

Definition at line 755 of file Chunk.cc.

◆ get_byte_order()

virtual std::string dmrpp::Chunk::get_byte_order ( )
inlinevirtual

Get the chunk byte order.

Definition at line 304 of file Chunk.h.

◆ get_bytes_read()

virtual unsigned long long dmrpp::Chunk::get_bytes_read ( ) const
inlinevirtual

Get the number of bytes read so far for this Chunk.

Definition at line 338 of file Chunk.h.

◆ get_curl_range_arg_string()

string dmrpp::Chunk::get_curl_range_arg_string ( )
virtual

Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream.

Definition at line 509 of file Chunk.cc.

◆ get_data_url()

std::shared_ptr< http::url > dmrpp::Chunk::get_data_url ( ) const
virtual

Get the data url for this Chunk's data block.

Definition at line 903 of file Chunk.cc.

◆ get_offset()

virtual unsigned long long dmrpp::Chunk::get_offset ( ) const
inlinevirtual

Get the offset to this Chunk's data block.

Definition at line 317 of file Chunk.h.

◆ get_position_in_array()

virtual const std::vector<unsigned long long>& dmrpp::Chunk::get_position_in_array ( ) const
inlinevirtual
Returns
The chunk's position in the array, as a vector of ints.

Definition at line 450 of file Chunk.h.

◆ get_rbuf()

virtual char* dmrpp::Chunk::get_rbuf ( )
inlinevirtual

Returns a pointer to the memory buffer for this Chunk. The return value is NULL if no memory has been allocated.

Definition at line 381 of file Chunk.h.

◆ get_rbuf_size()

virtual unsigned long long dmrpp::Chunk::get_rbuf_size ( ) const
inlinevirtual

Returns the size, in bytes, of the current read buffer for this Chunk.

Definition at line 442 of file Chunk.h.

◆ get_response_content_type()

virtual std::string dmrpp::Chunk::get_response_content_type ( )
inlinevirtual

Get the response type of the last response.

Definition at line 298 of file Chunk.h.

◆ get_size()

virtual unsigned long long dmrpp::Chunk::get_size ( ) const
inlinevirtual

Get the size of this Chunk's data block on disk.

Definition at line 309 of file Chunk.h.

◆ operator=()

Chunk& dmrpp::Chunk::operator= ( const Chunk rhs)
inline

I think this is broken. vector<Chunk> assignment fails in the read_atomic() method but 'assignment' using a reference works. This bug shows up in DmrppCommnon::read_atomic(). jhrg 4/10/18

Definition at line 288 of file Chunk.h.

◆ read_chunk()

void dmrpp::Chunk::read_chunk ( )
virtual

This method is for reading one chunk after the other, using a CURL handle from the CurlHandlePool.

Parameters
deflate
shuffle
chunk_size
elem_width

Definition at line 840 of file Chunk.cc.

◆ set_bytes_read()

virtual void dmrpp::Chunk::set_bytes_read ( unsigned long long  bytes_read)
inlinevirtual

Set the size of this Chunk's data block.

Parameters
sizeSize of the data in bytes

Definition at line 347 of file Chunk.h.

◆ set_data_url()

virtual void dmrpp::Chunk::set_data_url ( std::shared_ptr< http::url data_url)
inlinevirtual

Set the data url for this Chunk's data block.

Definition at line 330 of file Chunk.h.

◆ set_position_in_array() [1/2]

void dmrpp::Chunk::set_position_in_array ( const std::string &  pia)

parse the chunk position string

Extract information from the chunk position string and store as a vector of integers in the instance

Note
If we can change the DMR++ syntax to be less verbose and use a list of ints with whitespace as a separator, then the parsing code will be much simpler since istringstream is designed to deal with exactly that form of input.
Parameters
piaThe chunk position string. Syntax parsed: "[1,2,3,4]"

Definition at line 458 of file Chunk.cc.

◆ set_position_in_array() [2/2]

void dmrpp::Chunk::set_position_in_array ( const std::vector< unsigned long long > &  pia)

Set the chunk's position in the Array.

Use this method when the vector<unsigned long long> is known.

See also
Chunk::set_position_in_array(const string &pia)
Parameters
piaA vector of unsigned ints.

Definition at line 494 of file Chunk.cc.

◆ set_rbuf_to_size()

virtual void dmrpp::Chunk::set_rbuf_to_size ( )
inlinevirtual

Allocates the internal read buffer to be d_size bytes.

The memory of the read buffer is managed internally by this method.

The class maintains an internal flag, d_read_buffer_is_mine, which controls if the currently held read buffer memory is released (via a call to 'delete[]') when an this method is invoked.

If the CHunk owns the read buffer, then calling this method will release any previously allocated read buffer memory and then allocate a new memory block. The bytes_read counter is reset to zero.

Definition at line 366 of file Chunk.h.

◆ set_read_buffer()

void dmrpp::Chunk::set_read_buffer ( char *  buf,
unsigned long long  buf_size,
unsigned long long  bytes_read = 0,
bool  assume_ownership = true 
)
inline

Set the target read buffer for this chunk.

Parameters
bufThe new buffer to install into the Chunk.
buf_sizeThe size of the passed buffer.
bytes_readThe number of bytes that have been read into buf. In practice this is the offset in buf at which new bytes should be added, (default: 0)
assume_ownershipIf true, then the memory pointed to by buf will be deleted (using delete[]) when the Chunk object's destructor is called. If false then the Chunk's destructor will not attempt to free/delete the memory pointed to by buf. (default: true)

Definition at line 423 of file Chunk.h.

◆ set_response_content_type()

void dmrpp::Chunk::set_response_content_type ( const std::string &  ct)
inline

Set the response type of the last response.

Definition at line 301 of file Chunk.h.


The documentation for this class was generated from the following files: