One time huge import from Mongo to Solr in Centos 7

1. Install libcurl
yum install curl-devel

2. Install libxml
yum install libxslt-devel

2.1 Make sure both mongo & mongodb php extensions are installed
pecl install mongo
pecl install mongodb
Add the following to php.ini if not already added :
extension=mongo.so
extension=mongodb.so

3. Install solr php extension
pecl install -n solr

4. Enable solr extension
Add extension=solr.so in php.ini and restart apache
sudo service httpd restart
php -m | grep solr

Make sure there are no errors, if there are then it can be related to the order in which files are loaded. In which case remove `extension=solr.so` from php.ini and create a new file in /etc/php.d, possible 41-solr.ini such that it’s loaded at the end and add `extension=solr.so` in that.

Confirm if that file is being used by php.ini by the following command
php -i | grep “.ini”

6. Download scripts from https://github.com/5missions/mongoSolrImporter
Change the .ini file according to your mongo & solr connection settings.
Execute the following command to start the import from mongo to solr
php mongoSolrImporter.php -c mongoSolrImporter.ini

NOTE – The sample .ini file lists the field names in capitals, make sure to have the field name appropriate to your mongo collection. I have all the field-names in lower case in both mongo collection & solr document.

References : http://stackoverflow.com/a/28090368/566092 https://github.com/james75/SolrMongoImporter
https://bugs.php.net/bug.php?id=59267
https://github.com/5missions/mongoSolrImporter

Import huge mongo database from another server in the background

1. Run the following commands :
screen
mongo
use admin
db.runCommand({copydb:1,fromhost:’83.10.17.18:27017′,fromdb:’fromdbname’,todb:’todbname’})
ctrl + a + d (after pressing a in order to exit the screen)

2. Now exiting the SSH session, will still keep the command running. In order to get back to the screen, use the following command
screen -r

Reference : https://docs.mongodb.com/manual/reference/command/copydb/
http://www.tecmint.com/keep-remote-ssh-sessions-running-after-disconnection/
dsad

 

Setup laravel, mongo and solr website on centos 7 having apache and php56

1. Install git
yum install git

2. Install composer
curl -sS https://getcomposer.org/installer | php
chmod +x composer.phar
mv composer.phar /usr/local/bin/composer
composer -V
References : http://idroot.net/tutorials/how-to-install-php-composer-on-centos-7/
https://github.com/petehouston/laravel-deploy-on-shared-hosting

3. Install gcc
yum install gcc

4. Install mongo (php extension)
yum groupinstall “Development tools” -y
yum install openssl-devel
pecl install mongodb

5. Adding mongodb to php.ini
php -i | grep “Loaded Configuration File”
nano /etc/php.ini
Add the following line
extension=mongodb.so
/usr/sbin/apachectl restart
References : http://php.net/manual/en/mongodb.installation.pecl.php

6.
chmod -R o+w storage
composer install
composer dumpautoload -o
php artisan config:cache
php artisan route:cache
References : https://github.com/petehouston/laravel-deploy-on-shared-hosting

7.
php artisan cache:clear
sudo chmod -R 777 vendor storage
References : http://stackoverflow.com/a/38155938/566092

8. Redirect to /public using .htaccess

sudo nano /etc/httpd/conf/httpd.conf
Find the section <directory /var/www/html> and change AllowOverride None to AllowOverride All

<Directory /var/www/html>
AllowOverride All
</Directory>
Save and exit.

Create .htaccess in the root of your web directory with the following contents :

DirectorySlash Off
RewriteEngine on
RewriteRule ^(.*)$ public/$1

Now restart Apache to put the change into effect:
systemctl restart httpd
References : https://devops.profitbricks.com/tutorials/install-and-configure-mod_rewrite-for-apache-on-centos-7/

9. Creating Solr core
sudo -u solr bin/solr create -c video2
References : http://stackoverflow.com/a/36438016/566092

10.  Enabling gzip on apache
Add the following to .htaccess file

<ifmodule mod_deflate.c>
AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x- javascript application/javascript
</ifmodule>
Reference : http://stackoverflow.com/a/5846765/566092

How to setup centos 7 with apache, php56, mongo & solr

Step 1 of 5. Installing Nano

yum install nano

Step 2 of 5. Installing Apache

sudo yum install httpd
sudo systemctl start httpd.service
sudo systemctl enable httpd.service
Step 3 of 5. Installing PHP56
yum -y update
yum -y install epel-release
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
wget https://centos7.iuscommunity.org/ius-release.rpm
rpm -Uvh ius-release*.rpm
yum -y update
yum -y install php56u php56u-opcache php56u-xml php56u-mcrypt php56u-gd php56u-devel php56u-mysql php56u-intl php56u-mbstring php56u-bcmath
Step 4 of 5. Install MongoDB
nano vim /etc/yum.repos.d/mongodb.repo
Add the following content in the file
[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

Then exit and save the file with the command :wqyum -y update
yum -y install mongodb-org
systemctl start mongod
systemctl status mongodnano /etc/environment
add these lines:
LANG=en_US.utf-8
LC_ALL=en_US.utf-8mongostat (re-connect using ssh)
mongo (re-connect using ssh)

Reference : http://www.liquidweb.com/kb/how-to-install-mongodb-on-centos-7/
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/
http://stackoverflow.com/a/22430837/566092

Step 5 of 5. Installing ApacheSolrcd /tmp
wget http://www-us.apache.org/dist/lucene/solr/6.2.0/solr-6.2.0.tgz
tar xzf solr-6.0.1.tgz solr-6.0.1/bin/install_solr_service.sh –strip-components=2
sudo ./install_solr_service.sh solr-6.0.1.tgz
If the above command doesn’t work, edit install_solr_service.sh as follows :
Edit install_solr_service.sh, find the following line

adduser –system –shell /bin/bash –group –disabled-password –home “$SOLR_VAR_DIR” “$SOLR_USER”
and replace with
adduser –system –shell /bin/bash –home “$SOLR_VAR_DIR” “$SOLR_USER”
yum install lsof
service solr status

 

 

How to use Solarium (3.6.0) with Laravel 5.2

After hours of research falling into traps referring examples of different versions, finally managed to figure out the correct one.

1. Create `app/config/solr.php`

<?php

return array(
    'endpoint' => array(
        'key' => array(
            'host' => '127.0.0.1',
            'port' => 8983,
            'path' => '/solr/',
            'core' => 'mycore'
        )
    )
);

2. Add the following code to HomeController

<?php namespace App\Http\Controllers;  use Illuminate\Support\Facades\Config;  use Solarium\Core\Client\Client;  use Solarium_Exception;  class HomeController extends Controller {      protected $client;          public function __construct()     {          $configSolr = Config::get('solr');          $this->client = new Client($configSolr);
    }
public function index()
{
// create a ping query
$client = $this->client;
$ping = $client->createPing();
// execute the ping query
$result ='';
try {
$result = $client->ping($ping);
echo 'Ping query successful';
echo '<br?>';
var_dump($result->getData());
} catch (Solarium_Exception $e) {
echo 'Ping query failed';
var_dump($e);
}
return $result;
}
}

Fix an existing Data Scraper

Client has already got the scraper built from me some time ago and now he required a code-revision for adding more features.

The software scrapes a forum website to find out all the threads a particular user is involved in (has replied to). All thread urls are then written to an output txt file.

Job can be found here : https://www.elance.com/j/fix-an-existing-data-scraper/36333241/

Client Feedback : 

“Great freelancer! He is also communicative. “

Profit Collector (originally Golden Trader)

Profit Collector is set of services introduced from Golden Trader FX by applying the concept of social trading network.

By using Profit Collector, you will be able to do the following :
1 – Connect your MT4 and show your performance to the community.
2 – Analyze your performance with charts and graphs.
3 – Share your Ideas.
4 – Get more profit from your trading activities.

And MORE……….
Connect …… Trade…… Collect more profits .

Find it here

Ware X

Our warehouse management system includes the software, database, communications, security, administration and full vendor support. Along with completely integrated business-class components, we include consulting, installation and training, all with a single point of contact.

Features:-

Manages thousands of inbound suppliers.
Paperless Receiving and Picking.
Maintain customer and vendor databases
Items are tracked to a location
Maintains complete history of item life cycle
Inventory reports
Supports unlimited Sites, Locations, and Items
User Defined Fields – Facility to customize Location and item screens.
Database backup as per schedule.
User Manamgement and User Access Control.

link:- click here