Cascalog-core 2.0.0 API documentation

<-

macro

Constructs a query or predicate macro from a list of
predicates. Predicate macros support destructuring of the input and
output variables.

?-

Executes 1 or more queries and emits the results of each query to
the associated tap.

Syntax: (?- sink1 query1 sink2 query2 ...)  or (?- query-name sink1
query1 sink2 query2)

 If the first argument is a string, that will be used as the name
for the query and will show up in the JobTracker UI.

?<-

macro

Helper that both defines and executes a query in a single call.

Syntax: (?<- out-tap out-vars & predicates) or (?<- "myflow"
out-tap out-vars & predicates) ; flow name must be a static string
within the ?<- form.

??-

Executes one or more queries and returns a seq of seqs of tuples
 back, one for each subquery given.

Syntax: (??- query1 query2 ...) or (??- query-name query1 query2 ...)

If the first argument is a string, that will be used as the name
for the query and will show up in the JobTracker UI.

??<-

macro

Like ??-, but for ?<-. Returns a seq of tuples.

INumOutFields

IOutputFields

ISelectFields

aggregatefn

macro

aggregateop

bufferfn

macro

bufferiterfn

macro

bufferiterop

bufferop

cascalog-tap

combine

Merge the tuples from the subqueries together into a single
subquery. Doesn't ensure uniqueness of tuples.

compile-flow

Attaches output taps to some number of subqueries and creates a
Cascading flow. The flow can be executed with `.complete`, or
introspection can be done on the flow.

Syntax: (compile-flow sink1 query1 sink2 query2 ...)
or (compile-flow flow-name sink1 query1 sink2 query2)

 If the first argument is a string, that will be used as the name
for the query and will show up in the JobTracker UI.

construct

Parses predicates and output fields and returns a proper subquery.

cross-join

defaggregatefn

macro

Defines a filtering operation.

defaggregateop

macro

defbufferfn

macro

Defines a filtering operation.

defbufferiterfn

macro

Defines a filtering operation.

defbufferiterop

macro

defbufferop

macro

deffilterfn

macro

Defines a filtering operation.

deffilterop

macro

defmain

macro

Defines an AOT-compiled function with the supplied
`name`. Containing namespace must be marked for AOT compilation to
have any effect.

defmapcatfn

macro

Defines a mapcat operation.

defmapcatop

macro

defmapfn

macro

Defines a map operation.

defmapop

macro

defparallelagg

macro

Binds an efficient aggregator to the supplied symbol. A parallel
aggregator processes each tuple through an initializer function,
then combines the results each tuple's initialization until one
result is achieved. `defparallelagg` accepts two keyword arguments:

:init-var -- A var bound to a fn that accepts raw tuples and returns
an intermediate result; #'one, for example.

:combine-var -- a var bound to a fn that both accepts and returns
intermediate results.

For example,

(defparallelagg sum
:init-var #'identity
:combine-var #'+)

Used as

(sum ?x :> ?y)

defprepfn

macro

Defines a prepared operation.

div

Perform floating point division on the arguments. Use this instead
of / in Cascalog queries since / produces Ratio types which aren't
serializable by Hadoop.

explain

Explains a query (by outputting a DOT file).

outfile  - String location for DOT file output.
sink-tap - Sink tap for query. Shows on query explanation. Defaults to stdout if omitted.
query    - Query to be explained.

Syntax: (explain outfile query)  or (explain outfile sink query)

Ex: (explain "outfile.dot" (<- [?a ?b] ([[1 2]] ?a ?b)))

filterfn

macro

filterop

get-out-fields

Get the fields of a generator.

hfs-seqfile

Creates a tap on HDFS using sequence file format. Different
 filesystems can be selected by using different prefixes for `path`.

Supports keyword option for `:outfields`. See
`cascalog.cascading.tap/hfs-tap` for more keyword arguments.

 See http://www.cascading.org/javadoc/cascading/tap/Hfs.html and
 http://www.cascading.org/javadoc/cascading/scheme/SequenceFile.html

hfs-tap

Returns a Cascading Hfs tap with support for the supplied scheme,
opened up on the supplied path or file object. Supported keyword
options are:

`:sinkmode` - can be `:keep`, `:update` or `:replace`.

`:sinkparts` - used to constrain the segmentation of output files.

`:source-pattern` - Causes resulting tap to respond as a GlobHfs tap
when used as source.

`:sink-template` - Causes resulting tap to respond as a TemplateTap when
used as a sink.

`:templatefields` - When pattern is supplied via :sink-template,
this option allows a subset of output fields to be used in the
naming scheme.

See f.ex. the
http://docs.cascading.org/cascading/2.0/javadoc/cascading/scheme/local/TextDelimited.html
scheme.

hfs-textline

Creates a tap on HDFS using textline format. Different filesystems
can be selected by using different prefixes for `path`. Supported
keyword options are:

`:outfields` - used to select the fields written to the tap

`:compression` - one of `:enable`, `:disable` or `:default`

See `cascalog.cascading.tap/hfs-tap` for more keyword arguments.

See http://www.cascading.org/javadoc/cascading/tap/Hfs.html and
http://www.cascading.org/javadoc/cascading/scheme/TextLine.html

lfs-seqfile

Creates a tap that reads data off of the local filesystem in
 sequence file format.

Supports keyword option for `:outfields`. See
`cascalog.cascading.tap/lfs-tap` for more keyword arguments.

 See http://www.cascading.org/javadoc/cascading/tap/Lfs.html and
 http://www.cascading.org/javadoc/cascading/scheme/SequenceFile.html

lfs-tap

Returns a Cascading Lfs tap with support for the supplied scheme,
opened up on the supplied path or file object. Supported keyword
options are:

`:sinkmode` - can be `:keep`, `:update` or `:replace`.

`:sinkparts` - used to constrain the segmentation of output files.

`:source-pattern` - Causes resulting tap to respond as a GlobHfs tap
when used as source.

`:sink-template` - Causes resulting tap to respond as a TemplateTap
when used as a sink.

`:templatefields` - When pattern is supplied via :sink-template,
this option allows a subset of output fields to be used in the
naming scheme.

lfs-textline

Creates a tap on the local filesystem using textline format.

Supports keyword option for `:outfields`. See
`cascalog.cascading.tap/lfs-tap` for more keyword arguments.

 See http://www.cascading.org/javadoc/cascading/tap/Lfs.html and
 http://www.cascading.org/javadoc/cascading/scheme/TextLine.html

mapcatfn

macro

mapcatop

mapfn

macro

mapop

memory-source-tap

name-vars

normalize-sink-connection

num-out-fields

parallelagg

predmacro

macro

A more general but more verbose way to create predicate macros.

Creates a function that takes in [invars outvars] and returns a
list of predicates. When making predicate macros this way, you must
create intermediate variables with gen-nullable-var(s). This is
because unlike the (<- [?a :> ?b] ...) way of doing pred macros,
Cascalog doesn't have a declaration for the inputs/outputs.

See https://github.com/nathanmarz/cascalog/wiki/Predicate-macros

predmacro*

Functional version of predmacro. See predmacro for details.

prepfn

macro

Defines a prepared operation. Pass in an argument vector of two
items and return either a function or a Map with two
keywords; :operate and :cleanup

select-fields

Select fields of a named generator.

Example:
(<- [?a ?b ?sum]
    (+ ?a ?b :> ?sum)
    ((select-fields generator ["?a" "?b"]) ?a ?b))

sequence-file

stdout

Creates a tap that prints tuples sunk to it to standard
output. Useful for experimentation in the REPL.

text-line

to-tail

union

Merge the tuples from the subqueries together into a single
subquery and ensure uniqueness of tuples.

with-job-conf

macro

Modifies the job conf for queries executed within the form. Nested
with-job-conf calls will merge configuration maps together, with
innermost calls taking precedence on conflicting keys.

with-serializations

macro

Enables the supplied serializations for queries executed within the
form. Serializations should be provided as a vector of strings or
classes, like so:

(import 'org.apache.hadoop.io.serializer.JavaSerialization)
(with-serializations [JavaSerialization]
   (?<- ...))

Serializations nest; nested calls to with-serializations will merge
and unique with serializations currently specified by other calls to
`with-serializations` or `with-job-conf`.