Redundancy Amazon S3 / Rails

We use amazon S3 buckets for publishing js files and putting them on client web sites. In other words, js files are stored in amazon s3. The main problem is since these files are on client sites, we don't want to give the client new js codes if amazon s3 bucket goes down. Is there way in a Rails app to do the redundancy switch without giving client another code to install on their website? We want that piece of code to go use another bucket if the other one goes down without republishing anything or giving a new js code to the client. It seems impossible but I thought somebody might have an idea. Thanks in advance for all your help.

How to make my Java application scalable and fault tolerant?

In a simplified manner my Java application can be described as follows:

It is a web application running on a Tomcat server with a SOAP interface. The application uses JPA/Hibernate to store data in a MySQL database. The data stored consists of list of users, a list of hosts, and a list of URIs pointing to huge files (10GB) in the filesystem. The whole system consists of a central server, where my application is running on, and a bunch of worker hosts. A user can connect to the SOAP interface and ask the system to copy the files that belong to him to a specific worker host, where he then can analyze the data in some way (We cannot use NFS, we need to copy the data to the local disc storage of a worker host). The database then stores for each user on which worker host his files are stored.

At the moment the system is running with one central server with the Tomcat application and the MySQL database and 10 worker hosts and about 30 users which have 100 files (on average 10GB) size stored distributed over the worker hosts.

But in the future I have to scale the system by a factor of 100-1000. So I might have to deal with 10000 users, 100000 files and 10000 hosts. And the system should also become fault tolerant, so that I have don't have a single central server (which is the single point of failure in the system now), but maybe several ones. Also, if one of the worker hosts fails the system should be notified, so it doesn't try to copy files on that server.

My question is now: Which Java technologies could I use to make my application scalable and fault tolerant? What kind of architecture would you recommend? Should I still have a huge database storing all the information about all files, hosts and users in the system in one place, or should I better distributed my database on several hosts and synchronize them somehow?

How to apply different layouts to the same target in NLog?

NLog allows me to use SplitGroup to log my messages to several targets. I'd like to use this feature to log each message to a common, user-specific and date-specific logs at once:

<variable name="commonLog" value="${logDir}\Common.log" />
<variable name="username" value="${identity:fSNormalize=true:authType=false:isAuthenticated=false}" />
<variable name="userLog" value="${logDir}\ByUser\${username}.log" />
<variable name="dateLog" value="${logDir}\ByDate\${shortdate}.log" />

<target name="logFiles" xsi:type="SplitGroup">
  <target xsi:type="File" fileName="${commonLog}" layout="${myLayout}" />
  <target xsi:type="File" fileName="${userLog}" layout="${myLayout}" />
  <target xsi:type="File" fileName="${dateLog}" layout="${myLayout}" />
</target>

This is great, but I also want to use different layouts for different levels of severity. For example, errorLayout would include exception information and insert [!] marker so I could later highlight errors in log viewers like BareTail:

<variable name="stamp" value="${date} ${username} ${logger}" />

<variable name="debugLayout" value="${stamp} ... ${message}" />
<variable name="infoLayout" value="${stamp} [i] ${message}" /> 
<variable name="warnLayout" value="${stamp} [!] ${message}" />
<variable name="errorLayout"
   value="${warnLayout}${newline}${pad:padding=10:inner=${exception:format=ToString}}" />

<!-- logFiles target -->

<rules>
  <logger name="*" level="Debug" writeTo="logFiles" layout="debugLayout"  />
  <logger name="*" level="Info" writeTo="logFiles" layout="infoLayout" />
  <logger name="*" level="Warn" writeTo="logFiles" layout="warnLayout" />
  <logger name="*" level="Error" writeTo="logFiles" layout="errorLayout" />
</rules>

This code assumes Errors always come with exceptions and Warnings don't but that's not the point.

The problem is this configuration is wrong. It won't work because logger does not have layout attribute. It's defined for target only.

Layout which is being used must be declared by targets themselves but I see no means of specifying different layouts for different severity levels.

For now, I had to copy-paste the same configuration code four times just to have four different layouts for same set of files:

<targets>
  <target name="logFilesDebug" xsi:type="SplitGroup">
    <target xsi:type="File" fileName="${commonLog}" layout="${debugLayout}" />
    <target xsi:type="File" fileName="${userLog}" layout="${debugLayout}" />
    <target xsi:type="File" fileName="${dateLog}" layout="${debugLayout}" />
  </target>

  <target name="logFilesInfo" xsi:type="SplitGroup">
    <target xsi:type="File" fileName="${commonLog}" layout="${infoLayout}" />
    <target xsi:type="File" fileName="${userLog}" layout="${infoLayout}" />
    <target xsi:type="File" fileName="${dateLog}" layout="${infoLayout}" />
  </target>

  <target name="logFilesWarn" xsi:type="SplitGroup">
    <target xsi:type="File" fileName="${commonLog}" layout="${warnLayout}" />
    <target xsi:type="File" fileName="${userLog}" layout="${warnLayout}" />
    <target xsi:type="File" fileName="${dateLog}" layout="${warnLayout}" />
  </target>

  <target name="logFilesError" xsi:type="SplitGroup">
    <target xsi:type="File" fileName="${commonLog}" layout="${errorLayout}" />
    <target xsi:type="File" fileName="${userLog}" layout="${errorLayout}" />
    <target xsi:type="File" fileName="${dateLog}" layout="${errorLayout}" />
  </target>
</targets>

<rules>
  <logger name="*" level="Debug" writeTo="logFilesDebug"  />
  <logger name="*" level="Info" writeTo="logFilesInfo" />
  <logger name="*" level="Warn" writeTo="logFilesWarn" />
  <logger name="*" level="Error" writeTo="logFilesError" />
</rules>

This just hurts my eyes.
Is there any better way to do this and avoid duplication?

Avoid Redundancy in Python

I recently started using Python 2.6 for Ubuntu Server admin and have two minor issues concerning redundancy:

First thing are imports: They all look something like

import Class from Class

from class import Class

And the second thing are __init__ methods:

__init__(self,arg1,...,argn):
    self.arg1 = arg1
    ...
    self.argn = argn

Are there ways to avoid these duplications?

Clang 3.1 + libc++ Compile Error

I've built and installed (under the prefix ~/alt) LLVM-Clang trunk (23 apr 2012) successfully using GCC-4.6 on Ubuntu 12.04 and in turn libc++ using this Clang-build. When I want to use it I have to supply both -lc++ and -libstdc++ as follows

/home/per/alt/bin/clang -x c++ -I/home/per/alt/include/v1 -L/home/per/alt/lib -std=gnu++0x -g -Wall ~/f.cpp -lm -lc++ -lstdc++ -lpthread -o f

to compile f.cpp containing

#include <iostream>

using std::cout;
using std::endl;

int main(int argc, const char * argv[]) {
    cout << "sxx" << endl;
    return 0;
}

If I omit -lstdc++ I get the link error

/home/per/alt/include/v1/ostream:989: error: undefined reference to '__cxa_begin_catch'
/home/per/alt/include/v1/ostream:993: error: undefined reference to '__cxa_end_catch'
/home/per/alt/include/v1/ostream:993: error: undefined reference to '__cxa_end_catch'
/tmp/f-4l9mgl.o(.eh_frame+0xd3): error: undefined reference to '__gxx_personality_v0'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Compilation exited abnormally with code 1 at Tue Apr 24 13:59:22

Shouldn't libc++ be a full replacement for libstdc++?

How to use fastcgi_next_upstream in Nginx [closed]

I'd like to have 1 web server (nginx) and 2 FastCGI instances of the same application as back-end. The idea is to forward requests to second one if the first one is down.

Apparently, I need to use upstream and fastcgi_next_upstream. But I could not find a working example of a nginx.conf file. Does anybody have such example?

How to use code from one class in another? (Java)

I'm making a tank game and to avoid redundancy I'm making classes to extend. My MenuPanel looks like this atm (I've only written the code that matters for the question) (knop = dutch for button)

public class MenuPanel extends JPanel implements ActionListener
{

    private JButton playKnop, highScoreKnop, quitKnop, HTPKnop;
    private ImageIcon play, HS, quit, HTP;
    private Tanks mainVenster;

    public MenuPanel(Tanks mainVenster) 
    {
        this.mainVenster = mainVenster;
        this.setLayout(null); 

        int x = 95; 
        int width = 200; 
        int height = 50;    


        play = new ImageIcon(PlayPanel.class.getResource(/buttons/PLAY.png));
        playKnop = new JButton(play);
        playKnop.setBounds(x, y, width, height);
        playKnop.addActionListener(this);

        HS = new ImageIcon(PlayPanel.class.getResource(/buttons/HS.png));
        highScoreKnop = new JButton(HS);
        highScoreKnop.setBounds(x, 460, width, height);
        highScoreKnop.addActionListener(this);

        HTP = new ImageIcon(PlayPanel.class.getResource(/buttons/HTP.png));
        HTPKnop = new JButton(HTP);
        HTPKnop.setBounds(x, 515, width, height);
        HTPKnop.addActionListener(this);

        quit = new ImageIcon(PlayPanel.class.getResource(/buttons/QUIT.png));
        quitKnop = new JButton(quit);
        quitKnop.setBounds(x, 570, width, height);
        quitKnop.addActionListener(this);

        this.add(playKnop);
        this.add(quitKnop);
        this.add(HTPKnop);
        this.add(highScoreKnop);

        validate();
    }
}

because the code to make the buttons is exactly the same (except for the backgroundPath and the y-coordinate) I made a class button:

package menu;

import java.awt.Image;
import java.awt.event.ActionListener;

import javax.swing.ImageIcon;
import javax.swing.JButton;

public class button 
{


    public JButton button;
    public ImageIcon buttonImage;

    public int x = 95;
    public int width = 200;
    public int height = 50;

    public String backgroundPath;
    public int y;



    public button(String backgroundPath, int y)
    {
        this.backgroundPath = backgroundPath;
        this.y = y;

        buttonImage = new ImageIcon(PlayPanel.class.getResource(backgroundPath));
        button = new JButton();
        button.setBounds(x, y, width, height);;
        button.addActionListener(this); 
    }



}

that way, my menuPanel COULD look like this:

    package menu;

    @SuppressWarnings("serial")
    public class MenuPanel extends JPanel implements ActionListener
    {

    private button playKnop, highScoreKnop, quitKnop, HTPKnop;
    private JTextField naam;
    private Tanks mainVenster;



    public MenuPanel(Tanks mainVenster) 
    {
        this.mainVenster = mainVenster;
        this.setLayout(null);       

        playKnop = new button("/buttons/PLAY.png", 350);        
        highScoreKnop = new button("/buttons/HS.png", 460);
        quitKnop = new button("/buttons/QUIT.png", 515);
        HTPKnop = new button("/buttons/HTP.png", 570);


        this.add(playKnop);
        this.add(quitKnop);
        this.add(HTPKnop);
        this.add(highScoreKnop);


        validate();

    }


}

I don't know why, but when I do this the buttons won't appear, although the code from the button class is correct (bc when I use the code in the menuPanel itself it works)

I don't know how to fix this problem and I have multiple problems like this in my whole JavaProject, but if someone could explain me how to fix this I could get rid of all the redundancy in my project.

Thanks in advance, Lola

Thank you all so much! a combination of your answers helped me to let the buttons appear, but for some kind of reason the images don't load with them. my code now looks like:

public class MenuPanel extends JPanel implements ActionListener {

private Button playKnop, highScoreKnop, quitKnop, HTPKnop;

private Tanks mainVenster;

int x = 95, width = 200, height = 50;

public MenuPanel(Tanks mainVenster) 
{
    this.mainVenster = mainVenster;
    this.setLayout(null);

    playKnop = new Button("/buttons/PLAY.png", 350, this);      
    highScoreKnop = new Button("/buttons/HS.png", 460, this);
    quitKnop = new Button("/buttons/QUIT.png", 515, this);
    HTPKnop = new Button("/buttons/HTP.png", 570, this);

    this.add(playKnop);
    this.add(quitKnop);
    this.add(HTPKnop);
    this.add(highScoreKnop);

    validate();

}

public class Button extends JButton
{
    JButton button;
    ImageIcon buttonImage;

    String backgroundPath;
    int y;


    public Button(String backgroundPath, int y, MenuPanel menuPanel)
    {
        super();
        this.backgroundPath = backgroundPath;
        this.y = y;

        buttonImage = new ImageIcon(PlayPanel.class.getResource(backgroundPath));
        this.setBounds(x, y, width, height);;
        this.addActionListener(menuPanel); 
    }
}

}

(all the buttons do work!)

How to check for duplicate CSS rules?

I messed up my css and somehow i have a lot of the duplicate rules and my 1800 something lines css file is now of 3000+ lines..

Is there any way/tool that would take my css file as input and check for all the duplicate rules? and possibly generate a css removing those redundancies?

Generalizing these lines of code?

While programming, I ran into a wall with some code. It looks like this:

And that's the problem. I took a pretty screenshot to reduce my guilt. The pretty colors do not make up for the lack of maintainability. I have close to no idea how to generalize code like this.

What I tried?

Well, consider the periodicity of the 3rd and 6th arguments. It aligns with the periodicity of the other arguments too. Something like this would allow us to convert this code into a loop with 9 lines if we use an array. That's an improvement as we go down by 66%. However, that's not good enough. It would be best if this were changed to have 1 line. That would at least make it a bit more maintainable.

Is this really a problem?

Well, let's put it this way. That code up there may as well be wrong.

Cross-colo fail-over design, DNS level fail-over?

I'm interested in cross-colo fail-over strategies for web applications, such that if the main site fails users seamlessly land at the fail-over site in another colo.

The application side of things looks to be mostly figured out with a master-slave database setup between the colos and services designed to recover and be able to pick up mid-stream. I'm trying to figure out the strategy for moving traffic from the main site to the fail-over site. DNS failover, even with low TTLs, seems to carry a fair bit of latency.

What strategies would you recommend for quickly moving traffic between colos, assuming the servers at the main colo are unreachable?

If you have other interesting experience / words of wisdom about cross-colo failover I'd love to hear those as well.

programmatically optimizing expressions (by removing redundant computations)

I had a pretty big equation that I needed to use to solve for a given variable. So I used an online tool that was capable of rewriting an equation in terms of a given variable. It gave me some huge 700 character equation. I tested it, and it does work.

I can see some pretty obvious redundancies in the equation where it's recomputing a value that could be saved as a temporary variable instead. I could go through the entire equation and optimize it myself, but I'm likely to have to do this with many more equations, so I'd like to automate the process instead.

What are some good tools that will help optimize mathematical redundancies?
(It's just for a personal project, so I'd really prefer something free)

To all those people who I know will ask about this really being necessary: This is performance critical code, and from my experience, the AS3 compiler will not do these kind of optimizations on it's own. Removing redundancies will also make the code more readable.

Database normalization - who's right?

My professor(who claimed to have a firm understanding about systems development for many years) and I are arguing about the design of our database.

As an example: My professor insists this design is right: (list of columns)

Subject_ID
Description
Units_Lec
Units_Lab
Total_Units

etc...

Notice the total units column. He said that this column must be included. I tried to explain that it is unnecessary, because if you want it, then just make a query by simply adding the two.

I showed him an example I found in a book, but he insists that I dont have to rely on books too much in making our system. The same thing applies to similar cases as in this one:

student_ID
prelim_grade
midterm_grade
prefinal_grade
average

ect...

He wanted me to include the average! Anywhere I go, I can find myself reading articles that convince me that this is a violation of normalization. If I needed the average, I can easily compute the three grades. He enumerated some scenarios including ('Hey! What if the query has been accidentally deleted? What will you do? That is why you need to include it in your table!')

Do I need to reconstruct my database(which consists of about more than 40 tables) to comply with what he want? Am I wrong and just have overlooked these things?

EDIT:

Another thing is that he wanted to include the total amount in the payments table, which I believe is unnecessary(Just compute the unit price of the product and the quantity.). He pointed out that we need that column for computing debits and/or credits that are critical for the overall system management, that it is needed for balancing transaction. Please tell me what you think.

Sessions Failover with PHP-memcache against memcached

Colleagues!

I'm running php 5.3 (5.3.8) with memcache (2.2.6) client library (http://pecl.php.net/package/memcache) to deal with memcached server. My goal is to have failover solution for sessions engine, namely:

• Only native php sessions support (no custom handlers)
• Few memcached servers in the pool

What I expect is that in case if one of memcached servers is down, php will attempt to utilize the second server in the pool [will successfully connect it and become happy], however when first memcached server in the pool is down I'm receiving the following error:

Session start failed. Original message: session_start(): Server 10.0.10.111 (tcp 11211) failed with: Connection refused (111)

while relevant php settings are:

session.save_handler memcache
session.save_path tcp://10.0.10.111:11211?persistent=1&weight=1&timeout=1&retry_interval=10, tcp://10.0.10.110:11211?persistent=1&weight=1&timeout=1&retry_interval=10

and memcache settings (while I think that it's near to standard) are:

Directive   Local Value
memcache.allow_failover 1
memcache.chunk_size 8192
memcache.default_port   11211
memcache.default_timeout_ms 1000
memcache.hash_function  crc32
memcache.hash_strategy  standard
memcache.max_failover_attempts  20

Memcached still running on the second server and perfectly accessible from the WEB server:

telnet 10.0.10.110  11211
Trying 10.0.10.110...
Connected to 10.0.10.110 (10.0.10.110).
Escape character is '^]'.
get aaa
END
quit
Connection closed by foreign host.

So in other words, instead of querying all of the listed servers sequentially it crashes after unsuccessful attempt to connect the first server in the queue. Finally I do realize that there are releases of 3.0.x client library available, however it does not look too reliable for me as it still in beta version.

Please advice how can I get desired behavior with standard PHP, client lib and server.

Thanks a lot!

Best, Eugene

small code redundancy within while-loops (doesn't feel clean)

So, in Python (though I think it can be applied to many languages), I find myself with something like this quite often:

the_input = raw_input("what to print?\n")
while the_input != "quit":
    print the_input
    the_input = raw_input("what to print?\n")

Maybe I'm being too picky, but I don't like how the line the_input = raw_input("what to print?\n") has to get repeated. It decreases maintainability and organization. But I don't see any workarounds for avoiding the duplicate code without further worsening the problem. In some languages, I could write something like this:

while ((the_input=raw_input("what to print?\n")) != "quit") {
    print the_input
}

This is definitely not Pythonic, and Python doesn't even allow for assignment within loop conditions AFAIK.

This valid code fixes the redundancy,

while 1:
    the_input = raw_input("what to print?\n")
    if the_input == "quit":
        break
    print the_input

But doesn't feel quite right either. The while 1 implies that this loop will run forever; I'm using a loop, but giving it a fake condition and putting the real one inside it.

Am I being too picky? Is there a better way to do this? Perhaps there's some language construct designed for this that I don't know of?

Pretty print expression with as few parentheses as possible?

My Question: What is the cleanest way to pretty print an expression without redundant parentheses?

I have the following representation of lambda expressions:

Term ::= Fun(String x, Term t)
      |  App(Term t1, Term t2)
      |  Var(String x)

By convention App is left associative, that is a b c is interpreted as (a b) c and function bodies stretch as far to the right as possible, that is, λ x. x y is interpreted as λ x. (x y).

I have a parser that does a good job, but now I want a pretty printer. Here's what I currently have (pseudo scala):

term match {
    case Fun(v, t) => "(λ %s.%s)".format(v, prettyPrint(t))
    case App(s, t) => "(%s %s)".format(prettyPrint(s), prettyPrint(t))
    case Var(v)    => v
}

The above printer always puts ( ) around expressions (except for atomic variables). Thus for Fun(x, App(Fun(y, x), y)) it produces

(λ x.((λ y.x) y))

I would like to have

λ x.(λ y.x) y