Kyoto Cabinet: kyotocabinet::MapReduce Class Reference

Kyoto Cabinet

- Main Page
- Namespaces
- Classes
- Files

- Class List
- Class Index
- Class Hierarchy
- Class Members

Classes | Public Types | Public Member Functions | Protected Member Functions
kyotocabinet::MapReduce Class Reference

MapReduce framework. More...

`#include <kcdbext.h>`

List of all members.

## Classes
class | FlushThread
| Cache flusher.
class | MapChecker
| Checker for the map process.
class | MapVisitor
| Visitor for the map process.
struct | MergeLine
| Front line of a merging list.
class | ReduceTaskQueue
| Task queue for parallel reducer.
class | ValueIterator
| Value iterator for the reducer. More...
## Public Types
enum | Option { XNOLOCK = 1 << 0, XPARAMAP = 1 << 1, XPARARED = 1 << 2, XPARAFLS = 1 << 3, XNOCOMP = 1 << 8 }
| Execution options. More...
## Public Member Functions
| MapReduce ()
| Default constructor.
virtual | ~MapReduce ()
| Destructor.
virtual bool | map (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)=0
| Map a record data.
virtual bool | reduce (const char *kbuf, size_t ksiz, ValueIterator *iter)=0
| Reduce a record data.
virtual bool | preprocess ()
| Preprocess the map operations.
virtual bool | midprocess ()
| Mediate between the map and the reduce phases.
virtual bool | postprocess ()
| Postprocess the reduce operations.
virtual bool | log (const char *name, const char *message)
| Process a log message.
bool | execute (BasicDB *db, const std::string &tmppath="", uint32_t opts=0)
| Execute the MapReduce process about a database.
void | tune_storage (int32_t dbnum, int64_t clim, int64_t cbnum)
| Set the storage configurations.
void | tune_thread (int32_t mapthnum, int32_t redthnum, int32_t flsthnum)
| Set the thread configurations.
## Protected Member Functions
bool | emit (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)
| Emit a record from the mapper.

## Detailed Description

MapReduce framework.

Note:
  Although this framework is not distributed or concurrent, it is useful for aggregate calculation with less CPU loading and less memory usage.

## Member Enumeration Documentation

enum kyotocabinet::MapReduce::Option

Execution options.

Enumerator:

  XNOLOCK | avoid locking against update operations
  XPARAMAP | run mappers in parallel
  XPARARED | run reducers in parallel
  XPARAFLS | run cache flushers in parallel
  XNOCOMP | avoid compression of temporary databases

## Constructor & Destructor Documentation

kyotocabinet::MapReduce::MapReduce | ( |  | ) | `[explicit]`

Default constructor.

virtual kyotocabinet::MapReduce::~MapReduce | ( |  | ) | `[virtual]`

Destructor.

## Member Function Documentation

virtual bool kyotocabinet::MapReduce::map | ( | const char * | kbuf,
|  | size_t | ksiz,
|  | const char * | vbuf,
|  | size_t | vsiz
| ) |  | `[pure virtual]`

Map a record data.

Parameters:

  kbuf | the pointer to the key region.
  ksiz | the size of the key region.
  vbuf | the pointer to the value region.
  vsiz | the size of the value region.

Returns:
  true on success, or false on failure.

Note:
  This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::reduce | ( | const char * | kbuf,
|  | size_t | ksiz,
|  | ValueIterator * | iter
| ) |  | `[pure virtual]`

Reduce a record data.

Parameters:

  kbuf | the pointer to the key region.
  ksiz | the size of the key region.
  iter | the iterator to get the values.

Returns:
  true on success, or false on failure.

Note:
  To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::preprocess | ( |  | ) | `[virtual]`

Preprocess the map operations.

Returns:
  true on success, or false on failure.

Note:
  This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::midprocess | ( |  | ) | `[virtual]`

Mediate between the map and the reduce phases.

Returns:
  true on success, or false on failure.

Note:
  This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::postprocess | ( |  | ) | `[virtual]`

Postprocess the reduce operations.

Returns:
  true on success, or false on failure.

Note:
  To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::log | ( | const char * | name,
|  | const char * | message
| ) |  | `[virtual]`

Process a log message.

Parameters:

  name | the name of the event.
  message | a supplement message.

Returns:
  true on success, or false on failure.

bool kyotocabinet::MapReduce::execute | ( | BasicDB * | db,
|  | const std::string & | tmppath = `""`,
|  | uint32_t | opts = `0`
| ) |  |

Execute the MapReduce process about a database.

Parameters:

  db | the source database.
  tmppath | the path of a directory for the temporary data storage. If it is an empty string, temporary data are handled on memory.
  opts | the optional features by bitwise-or: MapReduce::XNOLOCK to avoid locking against update operations by other threads, MapReduce::XPARAMAP to run the mapper in parallel, MapReduce::XPARARED to run the reducer in parallel, MapReduce::XNOCOMP to avoid compression of temporary databases.

Returns:
  true on success, or false on failure.

void kyotocabinet::MapReduce::tune_storage | ( | int32_t | dbnum,
|  | int64_t | clim,
|  | int64_t | cbnum
| ) |  |

Set the storage configurations.

Parameters:

  dbnum | the number of temporary databases.
  clim | the limit size of the internal cache.
  cbnum | the bucket number of the internal cache.

void kyotocabinet::MapReduce::tune_thread | ( | int32_t | mapthnum,
|  | int32_t | redthnum,
|  | int32_t | flsthnum
| ) |  |

Set the thread configurations.

Parameters:

  mapthnum | the number of threads for the mapper.
  redthnum | the number of threads for the reducer.
  flsthnum | the number of threads for the internal flusher.

bool kyotocabinet::MapReduce::emit | ( | const char * | kbuf,
|  | size_t | ksiz,
|  | const char * | vbuf,
|  | size_t | vsiz
| ) |  | `[protected]`

Emit a record from the mapper.

Parameters:

  kbuf | the pointer to the key region.
  ksiz | the size of the key region.
  vbuf | the pointer to the value region.
  vsiz | the size of the value region.

Returns:
  true on success, or false on failure.