12:12 a.m.
Greetings All,
I have a few basic questions regarding the HSM framework in general and
regarding the "copytool" in particular:
1) If multiple "agents" are registered with the "coordinator", are the
HSM requests sent to all "agents" serially or in parallel? What is the
overall state of a file if some agents report success and other agents
report failure?
2) Using
https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifa...
as a reference, section 22.3 states "Only one copytool could be run by
agent node." If I'm backing up to Tape and NFS, I cannot run a "tape
copytool" and a "posix copytool" on the same agent node? Must I have 2
separate agent nodes, one for "tape copytool", and another for "posix
copytool"?
3) Using
https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifa...
as a reference, section 22.3.1 states "A Lustre file system supports an
unlimited number of copytool instances". Does it mean that I must have
unlimited agent nodes, each of which is running a single instance of
"copytool"?
4) When a copytool crashes, is there any mechanism within the HSM
framework to restart a copytool?
5) Is there any load-balancing capability with the HSM framework?
6) Is there HA capability with-in the HSM framework?
7) Is there a way to cluster copytool instances to get better throughput?
8) Is there a later version of the HSM manual than
https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifa...
? If so, where would I find it?
Thank you for your prompt reply.
Regards,
-aamir
5:23 p.m.
Hi Aamir,
Let me see if I can address some of your question inline.
Malcolm.
-----Original Message-----
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf
Of Aamir Rashid
Sent: Thursday, July 03, 2014 12:12 AM
To: hpdd-discuss(a)lists.01.org
Subject: [HPDD-discuss] Lustre 2.5.1. HSM
Greetings All,
I have a few basic questions regarding the HSM framework in general and
regarding the "copytool" in particular:
1) If multiple "agents" are registered with the "coordinator", are
the
HSM requests sent to all "agents" serially or in parallel? What is the
overall state of a file if some agents report success and other agents
report failure?
The coordinator acts like a job scheduler, queuing and dispatching requests to HSM agents.
Each request is sent to one of the available agents and the requests work at the
granularity of an individual file. This means that individual files are _not_ split up and
sent in parallel across multiple agents.
2) Using
https://build.hpdd.intel.com/job/lustre-
manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lustrehsm
as a reference, section 22.3 states "Only one copytool could be run by
agent node." If I'm backing up to Tape and NFS, I cannot run a "tape
copytool" and a "posix copytool" on the same agent node? Must I have 2
separate agent nodes, one for "tape copytool", and another for "posix
copytool"?
That is correct. Each agent runs one copytool, so if multiple copytools are required, they
must each run on their own machine.
3) Using
https://build.hpdd.intel.com/job/lustre-
manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lustrehsm
as a reference, section 22.3.1 states "A Lustre file system supports an
unlimited number of copytool instances". Does it mean that I must have
unlimited agent nodes, each of which is running a single instance of
"copytool"?
Yes, if you require an unlimited number of copytools, one must match those instances with
an unlimited supply of servers. The software does not impose a limit on the number of
instances but other considerations may force a practical limit :).
4) When a copytool crashes, is there any mechanism within the HSM
framework to restart a copytool?
No. Generally, a configuration management system (CF-Engine, Puppet, Chef) could be
employed to monitor and restart services if they fail. Intel Manager for Lustre also
monitors copytool instances and restarts them on fail.
5) Is there any load-balancing capability with the HSM framework?
The Coordinator will load balance requests across the copytool instances.
6) Is there HA capability with-in the HSM framework?
Lustre servers can be integrated into an HA cluster framework such as Pacemaker +
Corosync. The coordinator runs as a service thread on the metadata server and so will
failover if/when the MDT fails over. To provide HA in the HSM Agents, install multiple
agent servers, each with their own instance.
7) Is there a way to cluster copytool instances to get better
throughput?
Yes. Just launch multiple copytools, one per server. Each instance will automatically
register with the coordinator. The coordinator will automatically schedule work to the
pool of registered instances, similar to how a job scheduler operates.
8) Is there a later version of the HSM manual than
https://build.hpdd.intel.com/job/lustre-
manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lustrehsm
? If so, where would I find it?
I think that's the latest publically available documentation.
Thank you for your prompt reply.
Regards,
-aamir
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:06 a.m.
On 07/07/2014 03:23 AM, Cowe, Malcolm J wrote:
> 5) Is there any load-balancing capability with the HSM
framework?
The Coordinator will load balance requests across the copytool instances.
Hi Malcolm,
Thanks for your responses! However, I cannot see load-balancing in
affect at all. Does load balancing need to be explicitly turned on? If
so, how?
On my MDS, I've registered 2 agents, with same archive_id=5, each
running a single instance of posix copytool:
[root@hsm-mds1 lustre]# cat /proc/fs/lustre/mdt/lustrefs-MDT0000/hsm/agents
uuid=61f6bf53-2e22-d37b-45cd-fea234025701 archive_id=5
requests=[current:0 ok:11 errors:0]
uuid=516a502f-60d7-fa4e-09cf-ffe1ec43db3a archive_id=5
requests=[current:0 ok:0 errors:0]
Whenever I make a request to archive, the request goes to the *first*
registered one. As you can see from above, all 11 requests went to the
first one - thus no load balancing between the 2 agents.
If I kill the first copytool instance on agent 1, then the subsequent
archive requests will go to agent 2. However, that is not
load-balancing, it's more like failure recovery.
Please let me know how to correctly enable load-balancing, and/or what
is the load-balancing functionality available in 2.5.1.
Thanks,
-aamir
5:24 p.m.
Le 07/07/2014 20:06, Aamir Rashid a écrit :
Whenever I make a request to archive, the request goes to the *first*
registered one. As you can see from above, all 11 requests went to the
first one - thus no load balancing between the 2 agents.
Did you try to send
multiple request at the same time? Sending new
requests while copytool #1 is busy?
Aurélien
10:44 p.m.
Hi Aurelien,
Thank you for your response. Yes, I did as you stated below, but my
results are the same - all requests go to the first registered agent.
1) Is there any write-up that talks about HSM load-balancing? I have not
come across any documentation, going as far back as 2009. Furthermore,
load-balancing is a great feature to have, and products that have it, do
flaunt it. So, I find it unusual that no documentation talks about this
feature.
2) What is the load-balancing algorithm? Is it simply round-robin
between agents, or is it something more clever?
3) When multiple agents register for the *same* HSM back-end, must they
provide an explicit --archive-id, or should they let it default to ANY
(as shown below in my output from MDS)? BTW, I've tried it both ways
(with/without --archive-id and my results are the same)
4) Are there any throttles that control load-balancing? Something like
after 1G of data on agent-1, switch to agent-2 for the next 1G, and
flip-flop on 1G data transactions? Since my file sizes are small - 100M
each for testing - maybe I'm not hitting a threshold to kick start
load-balancing?
Here's some runtime data, that may help you figure out, if there's a
pilot error on my behalf. I issue two archive requests, one from each of
the 2 clients that I have, one after another. As the MDS results show
below, all requests are going to the first registered agent, even when
the first agent is busy and the second request comes in.
MDS
===
[root@hsm-mds1 lustre]# cat /proc/fs/lustre/mdt/lustrefs-MDT0000/hsm/agents
uuid=61f6bf53-2e22-d37b-45cd-fea234025701 archive_id=ANY
requests=[current:0 ok:23 errors:0]
uuid=516a502f-60d7-fa4e-09cf-ffe1ec43db3a archive_id=ANY
requests=[current:0 ok:0 errors:0]
Agent-1
=====
[root@hsm-client2 lustre_hsm]# /root/lhsmtool_posix --daemon --no-attr
--no-xattr --hsm-root /mnt/nfs/lustre_hsm1 --dry-run /mnt/lustre_hsm
[root@hsm-client gg]# lfs hsm_archive /mnt/lustre_hsm/gg/aa.5[1-9]
Agent-2
=====
[root@hsm-client lustre_hsm]# /root/lhsmtool_posix --daemon --no-attr
--no-xattr --hsm-root /mnt/nfs/lustre_hsm1 --dry-run /mnt/lustre_hsm
[root@hsm-client2 lustre_hsm]# lfs hsm_archive /mnt/lustre_hsm/gg/aa.6[1-9]
Thanks,
-aamir
On 07/08/2014 03:24 AM, DEGREMONT Aurelien wrote:
Le 07/07/2014 20:06, Aamir Rashid a écrit :
>
> Whenever I make a request to archive, the request goes to the *first*
> registered one. As you can see from above, all 11 requests went to
> the first one - thus no load balancing between the 2 agents.
Did you try to send multiple request at the same time? Sending new
requests while copytool #1 is busy?
Aurélien
2994
days inactive
3000
days old
4 comments
3 participants
participants (3)
-
Aamir Rashid -
Cowe, Malcolm J -
DEGREMONT Aurelien