by default bash shell allow limited files to open, since we had many index files, lucene has to open them up during performing search.
so when it exceeds the limit of 1025 files (which is default on our production environment). our application threw the following exception -
Caused by: java.io.FileNotFoundException: /var/indexes/ads-index/_gm.tvd (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:506)
at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:536)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:445)
at org.apache.lucene.index.TermVectorsReader.(TermVectorsReader.java:70)

after increasing this limit to 10,000 we didn’t find such problem exists. we had to apply “ulimit -n 10000″ command on bash shell to make it works.

TODO: possible bug, probable our system is opening up several index searcher.

on aawaj service they had  more than 150,000 items to index. we tried with our current release 0.5.1 to index all contents but we ended with extremely performance outage. later we released another version 0.5.2, where we added queued request handling and threw index optimization over an restful service uri – /rest/service/optimize/

here is the simple benchmark report -

version – 0.5.1 – first 100 items  ended in – 13.611 seconds.
version – 0.5.2 – first 100 items  ended in – 5.6152 seconds.

the change is really different and significant, later today we had anoter import on our repository, interestingly it took 1 hour to index 150,000 items. which was bit surprising since we were unable to do it with 0.5.1

actually we added single thread executor which keeps everything in queue and execute one by one. so we could remove  synchronized method.

here is an example code -

private final Executor mIndexTaskExecutor =

Executors.newSingleThreadExecutor();

public void addDocument(final Document pDocument) {

mIndexTaskExecutor.execute(

new Runnable() {

public void run() {

getLuceneIndexTemplate().addDocument(pDocument);

}

}

);

}

since i have been using IntellIJ idea for a long while, in GIT world i felt like alone though i was using idea but not getting any idea how to live without some IDE’s version control support.

i want to compare my changes before i commit, i want to see more user friendly diff merging tool which idea is providing with other version control support.

so i found GITIDEA over here http://github.com/Fudge/gitidea/tree/master, after cloning and building this project.

i found it wasn’t working. so looking into code i found, it is executing “git” command. since my “git” executable is not under /usr/s|bin, i had to create a new symblink to make it work.

now it is working great, i can see where i have changed just by looking at the code (which bhaves like the usual svn support).

thanks fudge

i have used this query – “price:[0 TO 2000]” to find all items which are bellow 2000.
but my search result was also including 12000 any clue ?

good news is i got my solution though it took bit longer to find my actual reason.

the quick answer is i have to use NumberTools.longToString(…) to index my number fields and to perform range search i have to convert my range value using NumberTools.

you can see my code here -
source view

just digging through lucene source code here is what i found -
to perform range query (which is later rewrite to range filter), it tooks lower token for example price: [0 TO 1000] here ’0′ is the lower token. it asks TermInfosReader to return a list of available terms for a specific field (such as price here).

RangeFilter iterate through each suggested term and seek for the token position in a random access segment file.

the problem is, the suggested terms are not sorted. also i didn’t find any way to sort them before filtering out.

though i index my content after sorting them properly. but it didn’t affect any result.

since lucene treats everything as string, and it uses String::compare to determine the range position, i think NumberTools, DateTools are the only way out there.

anyway, i am satisfy with what i got.
best wishes,
“…i do what i love to do…”

before digging inside the problem better i clarify my context -

i have been working with one of my projects, where i have a several configurations which suppose to be different in different deployment environment, for example – to make my service avail, user has to define his database, his content index directory, server port, ip address and few more.

all these configurations were kept within the spring bean declaration scope. though few of them were moved to the properties file through ext-util namespace handler.

previously to make those configuration changeable i kept those files under WEB-INF/classes so it overrides the default bundled resource.

but still it wasn’t user friendly, since my user has to change one xml document to add more index services also he has to edit different configurations from java properties file.

again when i started bundling jetty with my whole application i found it is tricker to keep those xml outside of application, since i don’t wanna make my user confused with lot of xml files. and i have to keep few xml out of  my classpath context, so spring has to load my configuration from my “config/index-configuration/” directory.

though by default spring can do it over “file:///absolute/path/to/file.xml” since spring is using java URL class to locate this resource.  but in my case i can’t put any aboslute path.
so i found a simple way out there to support relative path reference.

previously i had the following xml document -

now i have replaced them with the following change -

if you observe closely you will find “extfile:///” protocol reference, which is not standard url protocol. so to make it work i used the following bunch of code and configuration -

1. i have created a class “Handler” under the “com.ideabase.repository.core.protocols.extfile” package.
this package and class name is conventional, though you can change them lets sun explain about it.

2. i wrote the following code -

public class Handler extends URLStreamHandler {
protected URLConnection openConnection(final URL pURL) throws IOException {
final String path = pURL.getPath();
final URL resourceUrl;
if (path.startsWith(“/./”)) {
resourceUrl = new File(path.substring(1, path.length())).toURL();
} else {
throw new RuntimeException(“Unsupported url schema. currently you can ” +
“use only extfile:///./path/to/somewhere/in”);
}
return resourceUrl.openConnection();
}
}
3. i registered my protocol package through the system property -

System.setProperty(“java.protocol.handler.pkgs”, “com.ideabase.repository.core.protocols”);

now have a look on the main applicationContext.xml

if you observe closely again you could find the difference, on the normal case i have used relative path reference. so it must be from classloader or if it is loaded through file system based resource locator it will be located from the file system.

but now i can use cross context (relative file system and class loader path reference) resource file path reference. you can see i have imported “indexer-beans.xml” from the application relative path “config/index-configuration/…” where “data-access-layer-beans.xml” is loaded from the class loader.
now have a quick look on my configuration directory, to get an idea how i have improved configuration ability.

hope this will help you to solve similar problem.

assertNotNull(mProfileManager.authenticate(
Credential.Builder
.aNew().userName(userName).password(password).build()));

this type of scenario is common on most of the projects. managing different scoped configuration is one of the bothering stuff. so i was always thinking some simpler way with in spring container.

so i wrote “ext-util” namespace handler, which is locating property from multi scoped resource.
lets give you a more details use case -

in my test environment i have a property “index.directory=/Users/…/index”
in my test server i use “index.directory=/var/index”
in my production server i use “index.directory=/nfs/index”

here i have a bean which need this property.

you can see the hardcoded configuration. which is not easy to manage.
thats why here my “ext-util” goes with the simplest solution -

now my property resource is locating from “test” scope.

you might think, this “scope=’test’” is again hardcoded stuff, so i also support the following value -
scope=”sys:env”, now you have to define “-Denv=test” while you running your application, and spring bean will find the right property.

so what ever you put over your spring are no longer hardcoded as long as you locate them through “ext-util” which is stands for “extended utility”.

here is my attached source code – (i just extracted them from my on going project, thats why no build script or prebuilt package or maven artifact is bundled)

how to use -
include the source file with in your project
keep “META-INF” while you building your jar or keep them inside your classpath.
change your spring configuration -

xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xmlns:lang=”http://www.springframework.org/schema/lang”
xmlns:aop=”http://www.springframework.org/schema/aop”
xmlns:ext-util=”http://dev.somewherein.net/schema/ext-util”
xsi:schemaLocation=”http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://dev.somewherein.net/schema/ext-util http://dev.somewherein.net/resources/schemas/ext-util-0.1.xsd“/>
basic properties of ext-util -

1. from-property – which property you are looking for.

2. or-default – define default value if no property is found from default scope (system properties) this value is returned.

3. scope – define scope name, by default whatever you set except “sys:” prefixed string, will search for “classpath:”. also you can use “sys:abc” where abc is located from system properties.

4. path-prefix – you can define scope properties path prefix, this prefix can be “file:///, classpath:”. this value also can be prefixed by “sys:abc” where abc is located from system properties.

[download source code]
[download missing xsd file]

best wishes,

i was surprised while i had a look on his intelliJ idea color scheme, it all was dark, at the beginning i had a belief that dark color is stressful for eyes. recently when i got introduce with TextMate and their dark color scheme, i was really impressed.

so i didn’t make any delay to let harald know about my interest on his color scheme. he sent me his generated scheme for intelliJ idea. it was simply awesome.

here i have added screen snap of one of my ruby code in dark scheme. i have also attached my modified color scheme as well as the original one, which i received from harad.

modified intelliJ idea emacs color scheme

original intelliJ idea emacs color scheme by harald

few of my previously written posts -
usages of “final” keyword in java -

http://www.somewhereinblog.net/blog/hasan/28708715

usages of comment and coding
http://www.somewhereinblog.net/blog/hasan/28704020

let’s check out my recipes -

1. code from your peer developer perspective view.
while you writing your code, always think your code will be reviewed by some other developers, who is not like you,
who is very serious in simple fault and ignorance in writing code. your code will reflect who you are, so better be careful while you exposing yourselves (through writing code).

2. variable convention -
never write short variable name, always write variable which reflects it usages.
for example -

$user = new User() instead of $u = new User(). follow same convention when you have nested “loop”

3. source file naming convention (except java)
set your file name in such a way that will reflect the usages of the file.
for example -

verify_user_authentication.php (or in a packaging manner – user_util_verify_authentication.php). in java you better know how to write package.

4. keep your url domain centric
all urls should be inherited by some specific problem domain for example -

http://abc.com/user/login

http://abc.com/user/register

http://abc.com/user/update

here “user” is problem domain and “login|register|update” are actions or you could say problem.

5. method naming convention -
write method name in such a format that will reflect it usages. or simply make it self describing.
for example -

function verify_user_credentials() {}
def verify_whether_user_profile_is_complete() {}
private void checkUserFlyLimit() {}

6. don’t write query from controller:
build or execute query from DAO or helper or utility or domain itself. don’t put it over controller which limits the reuability and later extendability and interception

7. single concern
while coding, keep your method slim and less concerned about other implementation. for example -

private void isAuthorizedUser(pUser, pAction) {
if (userAlreadyLoggedOn(pUser) && userIsNotBlocked(pUser) && userHasPermission(pUser, pAction)) {
// boooooooooooooooomm…
}
}

8. think from testability
while writing your code always think from testable perspective view. this means, how you could testify your work or module
or bunch of code which your company is paying for.

9. follow coding norms -
i. think before coding

whatever you want to put on your code try to think or imagine before start coding.

ii. dump your think in comment

whatever you thought just write it down over comment or paper or in a common place where you or future developers could have look on. (better place in comment)

iii. write your algorithm in comment.

whatever you planned write in comment before kick start your coding.

iv. set up your unit test case.

find all probable assertion points from use case, put them over your unit test case.

v. write your code
vi. coding routine -

(1) test -> (2) code – go to > (1) [recursively]

10. always put your tag over the source file,
whatever you wrote, that reflects your confident and work efficiency, so better you tag your name over every source code you have written or modified. good for tech lead or project manager, because tagging each file with author name, it creates a hidden responsibility for the author good for both parties.
for example -

/**
* @author someone
*/

actually i have been using this project for my own development solution, so i belief this project will become a great strengthen feature gradually.

JETTY RUNNER is now running on max OSX, i have removed system try support in new tag v-0.2, soon i will release *.dmg package for mac osx. here are few screen snaps -

figure – 1: server console main window

figure – 2: global properties editor

change logs -
1. removed system tray support
2. removed default jmx configuration
3. added “start.sh” to launch JETTY RUNNER on *nix based platform where ruby script is installed.
here are few screen snaps, which i have taken from the newly added ruby script! -

figure – 3: newly added jetty runner on ruby

figure – 4: newly added jetty runner implementation in ruby

this script really great , at least i like it