What is a package

Packages are groups of actions that are used to perform specific processes on Red Sqirl. They generally run on the same platform e.g. Pig, Hive. We explain below the generalities. For more concrete example you can refer to the following sections:

Package Content

A Package will contain actions that can be used on Red Sqirl once it is installed.

Red Sqirl uses maven as a build manager. In addition to the Java code, Red Sqirl web application needs additional files to manage correctly a package. In our template package maven build, we gather those files in src/main/pck.

  • actions.txt: Contains the list of actions provided in this package, one action name per line.
  • settings.json: Lists all the settings the package requires. The settings can be organise in sub-categories.
  • lang.properties: Contains label and descriptions of all the settings the package needs.
  • package.properties: Contains the package name and the version property and a short description.
  • help: is a directory with an html file describing an action.
  • image: is a directory with a gif file for each action.

The Maven pom

In order to create a package, the first step is to set up a maven project with a unique group and artifact id.

A template of a pom file can be downloaded from here. You will then need to edit the pom with the necessary dependencies and configuration specific to your package.

        <properties>
                <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
                <generated.pck.dir>${project.build.directory}/${project.artifactId}-${project.version}/</generated.pck.dir>
                <resources.pck.dir>src/main/pck</resources.pck.dir>
                <maven.build.timestamp.format>yyyy-MM-dd</maven.build.timestamp.format>
                <timestamp>${maven.build.timestamp}</timestamp>
                <redsqirl.version>hadoop-2.7.0-hive-1.2.1--1.5</redsqirl.version>
        </properties>

        <pluginRepositories>
                <pluginRepository>
                    <id>archiva.idiro.com</id>
                    <name>Idiro Release Repository</name>
                    <url>http://archiva.idiro.com/repository/internal</url>
                </pluginRepository>
        </pluginRepositories>

        <repositories>
                <repository>
                    <id>archiva.idiro.com</id>
                    <name>Idiro Release Repository</name>
                    <url>http://archiva.idiro.com/repository/internal</url>
                </repository>
        </repositories>

        <reporting>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-project-info-reports-plugin</artifactId>
                                <version>2.6</version>
                        </plugin>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-javadoc-plugin</artifactId>
                                <version>2.9</version>
                                <configuration>
                                </configuration>
                        </plugin>
                </plugins>
        </reporting>

        <build>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-compiler-plugin</artifactId>
                                <version>3.1</version>
                                <configuration>
                                        <source>1.6</source>
                                        <target>1.6</target>
                                </configuration>
                        </plugin>

                        <!-- Copy Package property -->
                        <plugin>
                                <artifactId>maven-resources-plugin</artifactId>
                                <version>2.6</version>
                                <executions>
                                        <execution>
                                                <id>copy-resources</id>
                                                <phase>package</phase>
                                                <goals>
                                                        <goal>copy-resources</goal>
                                                </goals>
                                                <configuration>
                                                        <outputDirectory>${generated.pck.dir}</outputDirectory>
                                                        <resources>
                                                                <resource>
                                                                        <directory>${resources.pck.dir}</directory>
                                                                        <filtering>true</filtering>
                                                                </resource>
                                                        </resources>
                                                </configuration>
                                        </execution>
                                </executions>
                        </plugin>
                        
                        <!-- Copy the generated jar -->
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-dependency-plugin</artifactId>
                                <version>2.8</version>
                                <executions>
                                        <execution>
                                                <id>copy-artifact</id>
                                                <phase>package</phase>
                                                <goals>
                                                        <goal>copy</goal>
                                                </goals>
                                                <configuration>
                                                        <artifactItems>
                                                                <artifactItem>
                                                                        <groupId>${project.groupId}</groupId>
                                                                        <artifactId>${project.artifactId}</artifactId>
                                                                        <version>${project.version}</version>
                                                                        <type>${project.packaging}</type>
                                                                </artifactItem>
                                                        </artifactItems>
                                                        <outputDirectory>${generated.pck.dir}/lib/</outputDirectory>
                                                </configuration>
                                        </execution>
                                </executions>
                        </plugin>
                        <!-- Zip the package ready to be installed or uploaded on the analytics store -->
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-assembly-plugin</artifactId>
                                <version>2.4</version>
                                <executions>
                                        <execution>
                                                <id>zip_pck</id>
                                                <phase>package</phase>
                                                <goals>
                                                        <goal>single</goal>
                                                </goals>
                                                <configuration>
                                                        <finalName>${project.artifactId}-${project.version}</finalName>
                                                        <appendAssemblyId>false</appendAssemblyId>
                                                        <descriptors>
                                                                <descriptor>src/main/assembly/zip_pck.xml</descriptor>
                                                        </descriptors>
                                                </configuration>
                                        </execution>

                                </executions>
                        </plugin>
                </plugins>
        </build>

        <dependencies>
                <dependency>
                        <groupId>com.idiro.redsqirl</groupId>
                        <artifactId>redsqirl-workflow</artifactId>
                        <version>${redsqirl.version}</version>
                        <scope>compile</scope>
                </dependency>
                <dependency>
                        <groupId>junit</groupId>
                        <artifactId>junit</artifactId>
                        <version>4.11</version>
                </dependency>
                <dependency>
                        <groupId>log4j</groupId>
                        <artifactId>log4j</artifactId>
                        <version>1.2.12</version>
                        <scope>compile</scope>
                </dependency>
        </dependencies>

Settings

Every package can come with their own settings. They are described in a JSON file. The JSON file is composed of tabs and settings. You can also have template tabs to create several set of settings. See below an example.

{
  "settings": [
    {
      "property": "host_default",
      "scope": "any",
      "optional": "false",
      "default": "test",
      "type": "STRING"
    }
   ],
    "tabs": [
     {
      "template_name": "host",
      "settings": [
                            {
                              "property": "hostname",
                              "scope": "user",
                              "optional": "false",
                              "default": "dev.local.net",
                              "type": "STRING"
                            },
                            {
                              "property": "host_user",
                              "scope": "user",
                              "optional": "false",
                              "default": "igor",
                              "type": "STRING"
                            },
                            {
                              "property": "host_privateKeyFile",
                              "scope": "user",
                              "optional": "false",
                              "default": "/home/igor/.ssh/id_rsa",
                              "type": "STRING"
                            },
                            {
                              "property": "default_path",
                              "scope": "user",
                              "optional": "false",
                              "default": "/data/shiny-server",
                              "type": "STRING"
                            },
                            {
                              "property": "host_url",
                              "scope": "user",
                              "optional": "false",
                              "default": "http://dev:3838/",
                              "type": "STRING"
                            }
                        ]
                }
  ]
}

Lang file

The lang.properties file contain the description the label for each property in english.

redsqirl-shiny.host_default_label=host default
redsqirl-shiny.host_default_desc=default host

redsqirl-shiny.host.hostname_label=hostname
redsqirl-shiny.host.hostname_desc=hostname

redsqirl-shiny.host.host_user_label=host user
redsqirl-shiny.host.host_user_desc=host user

redsqirl-shiny.host.host_privateKeyFile_label=host private
redsqirl-shiny.host.host_privateKeyFile_desc=host private

redsqirl-shiny.host.default_path_label=default path
redsqirl-shiny.host.default_path_desc=default path

redsqirl-shiny.host.host_url_label=host url
redsqirl-shiny.host.host_url_desc=host url

Package Properties

The package properties contains four properties. nameToFooter is optional and set the name of the default package footer menu (no space or special charater accepted).

packageName=${project.artifactId}
version=${project.version}
description=My package description
nameToFooter=My-Package

The action related files

actions.txt file looks like below. The order will define the order of the auto-generated footer.

shiny_bar_plot
shiny_line_plot
shiny_pie_plot
shiny_build
deploy

The help and image directory would contain a file for each action. Also the image folder contain an image to represent the entire package. In our example the directories would have the following files.

deploy.html
shiny_bar_plot.html
shiny_build.html
shiny_line_plot.html
shiny_pie_plot.html
deploy.gif
redsqirl-shiny_package.gif
shiny_bar_plot.gif
shiny_build.gif
shiny_line_plot.gif
shiny_pie_plot.gif

A help file can contain the tags page1, page2, ..., pageN. Those tags will be used by the framework so that it scrolls to the relevant part of the documentation. Also note that to open links within the Red Sqirl web application page you need to follow a specific format.

<!DOCTYPE html>
<html>
<head>
<title>Shiny Build Help</title>
</head>

<body>
        <h1>Shiny Build</h1>
        <h2>Description</h2>

        <p>Create an application or add items to an existing application.</p>

        <h3>Quick Guide</h3>
        <p>
        <ul>
                <li>Input
                        <ul>
                                <li><a class="stopLink" id="helpmain_canvas"
                                        onclick="return false;" href="../help/canvas.html#output_type">
                                                Text or Binary Map Reduce
                                                Directory </a> or HDFS Text File</li>
                        </ul>
                </li>
                <li>Configurations
                        <ul>
                                <li>Choose the reference field</li>
                                <li>Choose the value fields</li>
                                <li>Choose the aggregation method</li>
                        </ul>
                </li>
                <li>Output
                        <ul>
                                <li>Zip File On HDFS</li>
                                <li>(Optional) Remote Directory</li>
                        </ul>
                </li>
        </ul>
        </p>

        <h2> <a id="page1">Application details - Page 1</a> </h2>


        <h2> <a id="page2">Plots - Page 2</a> </h2>

        <h2>Red Sqirl Shiny</h2>

        <p>Red Sqirl Shiny is a package that enables ploting functionality
        through Red Sqirl. It is powered by <a target="_blank"
            href="http://shiny.rstudio.com/" >Shiny</a> and <a target="_blank"
            href="https://cran.r-project.org/web/packages/googleVis/index.html"
            >googleVis</a>.</p>

</body>
</html>

How to deploy a package

  1. Log in to the http://marketplace.redsqirl.com
  2. Go to the Red Sqirl >> Package
  3. Click on the New Module button
  4. Fill out the form, give an image to your module and click Save
  5. On the new package line, click on "Versions" button
  6. Click on the New Version button
  7. Fill out the form and click save
    • The price field is not used at the moment so you can write "Free"
    • The type access is either PUBLIC, everybody can find and download your package, or PRIVATE, nobody can except you.
    • The file field will expect the zip file produced by the pom
    • If your package depends on another package, please specify in the "dependency" section

How to install a package

From Red Sqirl

  1. Log in to your Red Sqirl Instance
  2. Click on Software >> Install Package
  3. Choose the scope and click "Install"
  4. Log in with your analytics store details or create your account
  5. Search the package you wish to install
  6. Click install

Manually

  1. Log in to the http://marketplace.redsqirl.com
  2. Click on Red Sqirl >> Search
  3. Click on the package you wish to install
  4. Click on Request a key and specify the installation name
  5. Specify the scope (System means that is share by all users) and click 'OK'
  6. Download the zip file
  7. Download the license file (button on the bottom left of the module table)
  8. Connect Off-line with your user (have to be admin)
  9. Update the license file (licenseKey.properties)
  10. Upload the new package (zip file)