• Home
  • Shell
    • Emacs
    • Perl
    • screen
    • sed
  • Ubuntu
    • VNC
  • Web Development
    • Javascript
    • Joomla
    • MySQL
    • osTicket
  • Windows
    • Gimp

Posts tagged BASH

Usenet> Newznab backfilling

Jan31
2013
Written by Scott Rowley

I’ve recently been checking out newznab plus. I’ve found that while it’s nice as it slowly builds, it’s a bit annoying not having any older posts already loaded. Fortunately nn+ (nnp, newznab plus) comes with an option for backfilling. The instructions are useful but I’d like to expand on them to make it less of a hassle to update more than one particular group.

Please note that I’ve only been using nn+ for a few days now so there may indeed be easier ways to do things but this is the best way I’ve found so far. For example, I’m not exactly sure what the update_binaries_threaded.php file does but I don’t believe it does what I’m discussing here. I’m guessing from the “threaded” that it simply makes multiple threads of the same process to make the entire task faster.

The following is taken from http://newznab.readthedocs.org/en/latest/readme/#backfilling-groups

Backfilling Groups
Since most usenet providers have 800+ days of retention indexing all that information in one shot is not practical. Newznab provides a backfill feature that allow you to index past articles once your initial index has been built. To use the feature first set the back fill days setting in the group(s) to be backfilled to the number of day you wish to go back, making sure to set it higher than the number of days listed in the first post column. Once set run the backfill.php script in misc/update_scripts. Groups can be backfilled to a particular date using the script misc/update_scripts/backfill_date.php using the syntax:

php backfill_date.php 2011-05-15 alt.binaries.groupname.here

You can use the _threaded version of this script if on linux.

For more information on backfilling, see Update Scripts.

First off, what we need to do is to activate any groups we want. More than likely you’ve already done this. What you’ll want to do in order to backfill those groups is to change the ‘Backfill Days’ entry. This can be done through the webpage but if you are wanting to backfill all of them then this following command will make it much faster than manually clicking through a dozen or more groups.

You’ll need to be on the command line and then log into mysql

mysql -u root -p
Enter password:
mysql> use newznab; (or whatever your database name is)
mysql> update groups set backfill_target=365 where active=1;

You have now just changed all active groups to have a backfill of 365 days (1 year). Now we just need to run our backfill_date.php on each group. If we’ve got dozens of groups or more though this can be tedious to repeat over and over with each group. So instead we are going to get a list of all the active groups by running the following command:

mysql -u root -p newznab -e 'select name from groups where active=1;'> groups.txt

You can now go into the groups.txt file and remove and additional groups you want or just leave it intact if you want everything.

Finally we want to run the php backfill.php command to backfill each of our selected groups back 365 days (or whatever number you’ve selected, alternatively you can also skip updating the database with the amount of days to backfill and simply substitute the backfill_date.php as noted from the link.)

Normally this command would be run like so for a single group:

php backfill.php alt.binaries.groupname.here

But this is much too slow for multiple groups so we are going to use a ‘for’ function with our data from the groups.txt file we created.

for group in `cat groups.txt`; do php backfill.php $group; done

If you are like me and like to see your code before it runs then simply add an echo beforehand to view the results and then remove it when you are ready:

for group in `cat groups.txt`; do echo "php backfill.php $group"; done

The use of the backticks (`) around ‘cat groups.txt’ tells it to run the command in a subshell and then use the information from that back in the shell command.

Posted in PHP, Ubuntu - Tagged backfill, backfill days, for, groups, MySQL, newznab, newznab plus, nn+, nnp, php, update, usenet

MAN PAGE> ‘date’ (8.4)

Jan30
2013
Written by Scott Rowley

Note that this man page is regarding date version ‘date (GNU coreutils) 8.4’, your mileage may vary when using other versions.

I’m going to expand this section to include examples of every entry from the man page. Note that some examples include additional commands to demonstrate the information we are working with.

NAME
date — print or set the system date and time

SYNOPSIS
date [OPTION]… [+FORMAT]
date [-u|–utc|–universal] [MMDDhhmm[[CC]YY][.ss]]

DESCRIPTION
Display the current time in the given FORMAT, or set the system date.

-d, –date=STRING
display time described by STRING, not ‘now’

date
Wed Jan 30 10:08:18 CST 2013
date -d "1 year ago"
Mon Jan 30 10:08:33 CST 2012

-f, –file=DATEFILE
like –date once for each line of DATEFILE

cat years.txt
1 year ago
2 years ago
3 years ago
date -f years.txt
Mon Jan 30 10:19:17 CST 2012
Sun Jan 30 10:19:17 CST 2011
Sat Jan 30 10:19:17 CST 2010

READ MORE »

Tagged cat, date, example, Format, man, man page, page, shell, time

BASH> Colorize your ‘cat’ output

Apr07
2011
Written by Scott Rowley

Ok, well thats a bit misleading. We won’t actually be using ‘cat’ but ‘pygmentize’ and its not perfect, but its better than 1 single color

This runs on Debian/Gentoo based Linux systems

apt-get update
apt-get install pygmentize

Once thats been installed what I did was to make an alias in my .profile

alias pcat='pygmentize'

Make sure to reload your .profile if you’ve added your alias

cd ~
. .profile

Example cat and pcat outputs:
Cat OutputPygmentize Output

Tagged alias, cat, pcat, pygmentize, ubuntu

BASH> ‘find’ examples

Nov12
2010
Written by Scott Rowley

This will be a growing list as I find commands I need…

Find files of .ext that are 0 bytes and rm them.

find . -name "*.ext" -size 0 -exec rm {} \;
Tagged byte, example, find, rm, shell

BASH> Connect to another user’s console terminal using ‘screen’

Nov02
2010
Written by Scott Rowley

Recently, I was helping another Admin and I wanted to be able to share our screens but our IS department won’t allow for it. Being that we were working in a terminal session I decided to go this route instead.

Needed:
– Screen
– Local account on host computer/server for remote user

Install screen

sudo apt-get install screen

Set the screen binary (/usr/bin/screen) setuid root. By default, screen is installed with the setuid bit turned off, as this is a potential security hole.

sudo chmod +s /usr/bin/screen
sudo chmod 755 /var/run/screen

The host starts screen in a local xterm, using the command screen -S SessionName. The -S switch gives the session a name, which makes multiple screen sessions easier to manage.

screen -S screen-test

The remote user (remote_user) uses SSH to connect to the host computer (host_user).

ssh remote_user@server

The host (host_user) then has to allow multiuser access in the screen session via the command ^A :multiuser on (all ‘screen’ commands start with the screen escape sequence, ^A).

^A
:multiuser on

The host (host_user) must grant permission to the remote user (remote_user) to access the screen session using the command ^A :acladd user_name where user_name is the remote user’s login ID (remote_user).

^A
:acladd remote_user

The remote user can now connect to the hosts ‘screen’ session. The syntax to connect to another user’s screen session is screen -x host_user/sessionname.

screen -x host_user/screen-test
Posted in screen - Tagged connect, screen, share, terminal, user

BASH> How to install perl modules through CPAN on Ubuntu

Oct26
2010
Written by Scott Rowley

Install all dependent packages for CPAN

sudo apt-get install build-essential

Invoke the cpan command as a normal user

cpan

Once you hit on enter for “cpan” to execute, you be asked of some few questions. To make it simple for yourself, answer “no” for the first question so that the latter ones will be done for you automatically.

Enter the commands below

make install
install Bundle::CPAN

Now all is set and you can install any perl module you want. examples of what installed below:

cpan prompt> install IO::File
cpan prompt> install Net::SMTP_auth
cpan prompt> install Email::MIME::Attachment::Stripper
cpan prompt> install Mail::POP3Client
Tagged cpan, install, module, Perl

Emacs> Remove welcome screen/splash from emacs

Oct26
2010
Written by Scott Rowley

If you are tired of seeing the welcome screen, despite it being the 486th time you’ve opened emacs just perform the following:

Edit your .emacs file and add the following:

;; Remove splash screen
(setq inhibit-splash-screen t)
Posted in Emacs - Tagged Emacs, splash, welcome

BASH> Remove tabs from output

Oct26
2010
Written by Scott Rowley

In order to remove tabs from your output (say to get a specific variable) use the ‘tr’ command, for example

In my zone.file the line containing “Serial” looks like the following:
If I performed

grep Serial zone.file

I would get the following output

 2009110904 ; Serial (yyyymmdd##)

You can’t see them here but there are several tabs before that date. In my example I want to get just the actual serial number (2009110904). To do this I’ll first strip off the “; Serial…” information;

grep Serial zone.file | awk -F; {'print $1'}

This command is telling it to look for the ; special character and break up the output there, then from there it is told to print the first column of information this results in:

 2009110904

The problem here is that if we want this to be our variable we are actually working with ” 2009110904″ instead of the desired “2009110904” In this particular case we know those are tabs not spaces. So we’ll need to use tr in order to remove those tabs:

grep Serial zone.file | awk -F\; {'print $1'} | tr -d 't'

This will return to us exactly what we want

2009110904
Tagged awk, grep, tab, tr

BASH> Hello World Script

Oct26
2010
Written by Scott Rowley

So you’ve either recently come to the Unix world or you are finally getting around to some shell scripting to ease your daily operations (Say hi Jim!). So for starters what *IS* bash scripting? Scripting in general is making a program or “script” do all or most of your work for you. For instance if you wanted to find every instance of the word “hosting” in a particular file that is 30,000 lines long. That would take a LONG time doing that by hand and you would likely miss several along the way (being a fallible human and all). So instead we would use a grep (or similiar command) from the command line.
Well that may be easy enough for just one command but what if we then need to check all of those lines that are found and seperate them by date or some other criteria, then sort them out etc? For this we will use scripting. We can also use scripting for events that we want to take place every single day, whether it be simple or complex. This can be done via a cronjob, but we can get to that in a later article. So first off we’ll need to find where our “bash” resides…

which bash > helloWorld.sh

“which bash” will give us the location of the bash that is being used within our path. The > will send that result into a new file titled “helloWorld.sh” that we will use for our script. Now if we open up that file we will have something like the following:

/usr/bin/bash

Nothing else should be in our file at this point. This is good but not quite what we need. We need to add #! in front of this. This tells our script that we will be using /usr/bin/bash as our shell to run the rest of this script.

#!/usr/bin/bash

To complete our very first script all we’ll need to do is enter in the following a few lines (doesn’t matter how many) after our #!/usr/bin/bash line, like so:

#!/usr/bin/bash
echo "Hello World!"

Now we have our script ready, we just need to execute it. There are two ways to accomplish this, first we can simply chmod the file to give it the execute (x) permission. For simplicity sake we’ll say its:

chmod 755 helloWorld.sh

The 7 sets it so that the owner (you) can do anything with the file, the first 5 sets it so that anyone in its group can read and execute it and likewise for the last 5 but for this is for everyone, not just the group. More on chmod and chown in future articles.

Now that we have our file chmod’d to be executable we can run it right from where we are by typing

./helloWorld.sh

and our output should be simply:

Hello World!

As mentioned there are two ways to execute the file. For the second method we could type the following and not have to worry about permissions (only ownership — but since we own it we really don’t have to worry about that either).

/usr/bin/bash ./helloWorld.sh

And, again, our output would be the same:

Hello World!

Congratulations, you’ve just made your first Bourne Again Shell (Bash) Script.

Tagged chmod, command, echo, hello world, script

BASH> Get all but the last field

Oct26
2010
Written by Scott Rowley

Recently I had need of cutting up domain names such as domain.com, this is easy enough for as simple a domain as this. However, the problem arises when you need to cut up something more like “education.k12.ia.us”. In order to get the extension alone in either case you can do the following:

echo domain.com | awk -F. {'print $2'}
com

Now normally if you have a domain as simple as example.com with only one period then you can can just do the opposite of this and print $1:

echo domain.com | awk -F. {'print $1'}
domain

The problem with more complex URLs is that awk’s field delimiter is now looking at multiple periods to look at, could be just one or it could be two, three, four or more.

To get the final extension in this case then we’ll use the following:

echo education.k12.ia.us | awk -F. {'print $NF'}
us

Then for the rest of it we’ll have to reverse it, then cut off the first field then reverse it again:

echo education.k12.ia.us | rev | cut -d. -f2- | rev
education.k12.ia
Tagged awk, delimiter, domain, extension, field
« Older Entries

Corrections? Questions? Comments?

Find an error?
Everything work out great for you?
Have some feedback?
Like to see something added to the article?

PLEASE leave us a comment after the article and let us know how we are doing, or if something needs corrected, improved or clarified.

Thank you!
- The Management

Advertisement

Sudo Bash
By Geeks - For Geeks

Back to Top