Showing posts with label apache. Show all posts

Wednesday, 12 July 2017

Create and Run Apache Spark Scala project using Eclipse and SBT

This post shows a simple way to create and run a apache spark scala project using eclipse and SBT. Following this post at link (http://czcodezone.blogspot.sg/2015/11/create-scala-project-using-scala-ide.html) to create a SBT-compatiable scala project in Scala IDE, open the build.sbt in the project and add the following line towards the end of the file.

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"

Further more, since Spark needs to run a spark cluster in a jar, a sbt plugin must be added for the packaging. Create a file named "assembly.sbt" in the "project" folder under the root directory and put the following content:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

Now in the command line, navigate to the project root directory and run "sbt compile", "sbt package" command after you put in your spark scala code, then "sbt run" or "spark-submit" depending on whether you want to run it locally or submit to spark cluster

{ Read More }

Tuesday, 9 May 2017

Creating a custom 404 Not Found page in Apache Server

DEMO
You might have seen a 404 page on different sites such as Google,Yahoo etc... You can make this on your website too if you are using Apache Server. The default 404 page will be like this :

Its just simple plain HTML 404 page and its ugly. To Change the default 404 page Do as below.

Go to your sites folder.

Create a file named .htaccess (just .htaccess no name only file extension).

Open the file and add the line.

ErrorDocument 404 /pathtofile

Replace /pathtofile with your 404 page file name which should be situated in the same folder as the file .htaccess . The custom 404 file can be in PHP or HTML.
You can also add HTML to it like this:

ErrorDocument 404 <html><head></head><body><h1>404 Not Found-Subins Blog</h1></body></html>

You dont have to reload Apache Server ! Its finished. Test it out. Heres an example of a custom 404 page.

{ Read More }

Monday, 17 April 2017

Count Frequency Of Values In A Column Using Apache Pig

There may be situations to count the occurence of a value in a field.
Let this be the sample input bag.

user_id course_name user_name
1 Social Anju
2 Maths Malu
1 English Anju
1 Maths Anju

Say we need to calculate no of occurence of each user_name.

Anju 3
Malu 1

Inorder to achieve this - COUNT Built In Function can be used.

COUNT Function in Apache Pig

COUNT function compute the number of elements in a bag.
To group count a preceding GROUP BY statement and for global counts GROUP ALL statement is required.

The basic idea to do the above example is to group by user_name and count the tuples in the bag.

--count.pig

 userAlias = LOAD /home/sreeveni/myfiles/pig/count.txt as 
 (user_id:long,course_name:chararray,user_name:chararray);
 groupedByUser = group userAlias by user_name;
 counted = FOREACH groupedByUser GENERATE group as user_name,COUNT(userAlias) as cnt;
 result = FOREACH counted GENERATE user_name, cnt;
 store result into /home/sreeveni/myfiles/pig/OUT/count;

The COUNT function ignores NULLs, that is tuple in the bag will not be counted if the first field in this tuple is NULL.
COUNT_STAR can be used to count fields including NULL values.

{ Read More }

Wednesday, 12 July 2017

Create and Run Apache Spark Scala project using Eclipse and SBT